jaegertracing/jaeger · error

failed to parse bulk response: %w

Error message

failed to parse bulk response: %w

What it means

After a successful HTTP round-trip, sendChunk JSON-decodes the _bulk response body into bulkResponse. This error means the body the client received was not the expected JSON shape ({"items":[...]}), so per-item durability cannot be determined. The chunk is reported as 0 durable and the caller retries.

Source

Thrown at internal/storage/elasticsearch/esclient/sync_bulk.go:182

			w.metrics.LatencyOk.Record(time.Since(start))
		} else {
			w.metrics.LatencyErr.Record(time.Since(start))
		}
	}()

	raw, err := w.client.request(ctx, elasticRequest{
		endpoint:    "_bulk",
		method:      http.MethodPost,
		body:        body,
		contentType: "application/x-ndjson",
	})
	if err != nil {
		return 0, fmt.Errorf("bulk request failed: %w", err)
	}

	var resp bulkResponse
	if err := json.Unmarshal(raw, &resp); err != nil {
		return 0, fmt.Errorf("failed to parse bulk response: %w", err)
	}
	// A well-formed _bulk response reports exactly one result per document, in
	// request order. If a proxy or partial response returns fewer (or more), the
	// per-item accounting below can't be trusted, so fail the whole chunk (0
	// durable) rather than silently miscount — the caller retries the batch.
	if len(resp.Items) != count {
		return 0, fmt.Errorf("malformed bulk response: %d item results for %d documents", len(resp.Items), count)
	}
	// The HTTP round-trip and response parsing succeeded, so record latency-ok even
	// when some items are rejected below. This matches the async BulkIndexer, whose
	// latency-err covers only a whole-request (transport/non-2xx) failure while
	// per-item rejections are reflected through the errors counter alone; latency-err
	// therefore keeps its meaning of "the request failed", not "some items failed".
	success = true
	// Derive failures from the per-item statuses, not the top-level `errors` flag:
	// a malformed or proxied response could report errors:false while an item still
	// carries a failing status, and silently succeeding there would advance the
	// Kafka offset over lost data — exactly what this synchronous writer prevents.

View on GitHub (pinned to 806f444784)

Solutions

1. Curl the configured server URL for /_bulk or / to see what is actually answering; fix the URL or remove the intercepting proxy.
2. Ensure no proxy rewrites/intercepts the response body; check proxy error pages in the path.
3. Verify the endpoint answers with ES/OS JSON: curl -s http://host:9200 | head and confirm the version JSON.
4. Check for TLS-terminating layers returning plaintext HTML errors and correct certs/ports.

Example fix

// before: URL points at a proxy UI that returns HTML
servers: ["http://proxy.internal/kibana"]
// after: point at the ES data node/API
servers: ["http://es-client-http.logging:9200"]

// confirm the endpoint speaks Elasticsearch JSON before sending data
resp, err := http.Get(server + "/")
if err == nil {
    var v map[string]any
    if json.NewDecoder(resp.Body).Decode(&v) != nil || v["cluster_name"] == nil {
        log.Fatalf("%s is not an Elasticsearch/OpenSearch endpoint", server)
    }
    resp.Body.Close()
}

// probe the raw response shape before trusting per-item data
var probe struct {
    Items []json.RawMessage `json:"items"`
}
if json.Unmarshal(raw, &probe) != nil || len(probe.Items) == 0 {
    // response body is not an ES bulk response — treat as transport/intermediary failure
}

err := writer.WriteBatch(ctx, items)
if err != nil && strings.Contains(err.Error(), "failed to parse bulk response") {
    // body wasn't ES JSON: check proxies/gateways before retrying
    log.Errorf("non-JSON response from ES path: %v", err)
}

When it happens

Trigger: The client.request call returns raw bytes that json.Unmarshal cannot decode into bulkResponse — e.g. an HTML error page from a proxy, a truncated body, a gzip/encoding mismatch, or an auth gateway returning non-JSON on a 200-level or odd response.

Common situations: A reverse proxy or load balancer (nginx, Istio, AWS ALB) intercepting the request and returning an HTML 502/401 page; a captive portal; a broken intermediary stripping Content-Encoding; hitting a non-Elasticsearch endpoint at the configured URL.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• malformed bulk response: %d item results for %d documents
• unmarshalling ElasticSearch documents failed
• bulk request failed: %w
• %d of %d bulk items rejected: %s
• unmarshalling documents failed: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/874a940a147c6070. Report an issue: GitHub.