jaegertracing/jaeger · error

malformed bulk response

Error message

malformed bulk response: %d item results for %d documents

What it means

A well-formed _bulk response contains exactly one result per submitted document, in request order. When the parsed response has a different number of item entries than documents sent, the per-item accounting cannot be trusted, so sendChunk fails the entire chunk (0 durable) rather than silently miscounting — the caller retries the batch.

Solutions

1. Bypass the intermediary by pointing the servers list directly at Elasticsearch/OpenSearch nodes and re-test.
2. Inspect the actual response body (tcpdump, proxy logs, or a snapshottest-style recorder) to find which middlebox alters it.
3. If a custom proxy is required, configure it to pass the _bulk response through unmodified.
4. Report/fix the compatibility layer so it returns one result item per document in order.

// validate that the configured endpoint is a real ES/OS node, not a
// custom gateway that may rewrite _bulk responses
resp, _ := http.Get(server + "/")
b, _ := io.ReadAll(resp.Body)
if !bytes.Contains(b, []byte("cluster_name")) {
    log.Fatalf("endpoint %s does not look like Elasticsearch", server)
}

// pre-flight shape check equivalent to the client's own invariant
func looksLikeBulkResponse(raw []byte, count int) bool {
    var r struct{ Items []json.RawMessage `json:"items"` }
    if json.Unmarshal(raw, &r) != nil { return false }
    return len(r.Items) == count
}

if err := writer.WriteBatch(ctx, items); err != nil {
    var malformed = strings.Contains(err.Error(), "malformed bulk response")
    if malformed {
        // response item count != sent docs: an intermediary is rewriting
        // bodies; fail loudly / bypass proxy rather than blind-retrying
    }
    return err
}

When it happens

Trigger: A proxy or intermediary returns a truncated or aggregated response; a custom gateway strips or merges the items array; a software/hardware load balancer returns its own partial body; essentially any middlebox that alters the _bulk response body while keeping it JSON-parseable.

Common situations: Corporate HTTP proxies that rewrite or buffer responses; misconfigured service meshes retrying and merging bodies; custom ES gateways (security or rate-limiting proxies) returning synthetic JSON; running against a mock/compatibility layer that does not implement _bulk faithfully.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to parse bulk response
• bulk request failed
• of bulk items rejected
• aggregation bucket has a non-string key
• at least one storage backend is required

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/2815d2a2ce0c8d39. Report an issue: GitHub.

Appendix: source

Thrown at internal/storage/elasticsearch/esclient/sync_bulk.go:189

		endpoint:    "_bulk",
		method:      http.MethodPost,
		body:        body,
		contentType: "application/x-ndjson",
	})
	if err != nil {
		return 0, fmt.Errorf("bulk request failed: %w", err)
	}

	var resp bulkResponse
	if err := json.Unmarshal(raw, &resp); err != nil {
		return 0, fmt.Errorf("failed to parse bulk response: %w", err)
	}
	// A well-formed _bulk response reports exactly one result per document, in
	// request order. If a proxy or partial response returns fewer (or more), the
	// per-item accounting below can't be trusted, so fail the whole chunk (0
	// durable) rather than silently miscount — the caller retries the batch.
	if len(resp.Items) != count {
		return 0, fmt.Errorf("malformed bulk response: %d item results for %d documents", len(resp.Items), count)
	}
	// The HTTP round-trip and response parsing succeeded, so record latency-ok even
	// when some items are rejected below. This matches the async BulkIndexer, whose
	// latency-err covers only a whole-request (transport/non-2xx) failure while
	// per-item rejections are reflected through the errors counter alone; latency-err
	// therefore keeps its meaning of "the request failed", not "some items failed".
	success = true
	// Derive failures from the per-item statuses, not the top-level `errors` flag:
	// a malformed or proxied response could report errors:false while an item still
	// carries a failing status, and silently succeeding there would advance the
	// Kafka offset over lost data — exactly what this synchronous writer prevents.
	out := resp.classify()
	if out.conflicts > 0 {
		// Counted in the durable total returned below, mirroring the async indexer's
		// onItemConflict: an already-present document achieved the write's goal.
		w.logger.Debug("bulk items already present (idempotent write)",
			zap.Int("conflicts", out.conflicts), zap.Int("total", count),
			zap.Int("status", http.StatusConflict))

View on GitHub (pinned to 806f444784)