jaegertracing/jaeger · error

malformed bulk response: %d item results for %d documents

Error message

malformed bulk response: %d item results for %d documents

What it means

A well-formed _bulk response contains exactly one result per submitted document, in request order. When the parsed response has a different number of item entries than documents sent, the per-item accounting cannot be trusted, so sendChunk fails the entire chunk (0 durable) rather than silently miscounting — the caller retries the batch.

Source

Thrown at internal/storage/elasticsearch/esclient/sync_bulk.go:189

		endpoint:    "_bulk",
		method:      http.MethodPost,
		body:        body,
		contentType: "application/x-ndjson",
	})
	if err != nil {
		return 0, fmt.Errorf("bulk request failed: %w", err)
	}

	var resp bulkResponse
	if err := json.Unmarshal(raw, &resp); err != nil {
		return 0, fmt.Errorf("failed to parse bulk response: %w", err)
	}
	// A well-formed _bulk response reports exactly one result per document, in
	// request order. If a proxy or partial response returns fewer (or more), the
	// per-item accounting below can't be trusted, so fail the whole chunk (0
	// durable) rather than silently miscount — the caller retries the batch.
	if len(resp.Items) != count {
		return 0, fmt.Errorf("malformed bulk response: %d item results for %d documents", len(resp.Items), count)
	}
	// The HTTP round-trip and response parsing succeeded, so record latency-ok even
	// when some items are rejected below. This matches the async BulkIndexer, whose
	// latency-err covers only a whole-request (transport/non-2xx) failure while
	// per-item rejections are reflected through the errors counter alone; latency-err
	// therefore keeps its meaning of "the request failed", not "some items failed".
	success = true
	// Derive failures from the per-item statuses, not the top-level `errors` flag:
	// a malformed or proxied response could report errors:false while an item still
	// carries a failing status, and silently succeeding there would advance the
	// Kafka offset over lost data — exactly what this synchronous writer prevents.
	out := resp.classify()
	if out.conflicts > 0 {
		// Counted in the durable total returned below, mirroring the async indexer's
		// onItemConflict: an already-present document achieved the write's goal.
		w.logger.Debug("bulk items already present (idempotent write)",
			zap.Int("conflicts", out.conflicts), zap.Int("total", count),
			zap.Int("status", http.StatusConflict))

View on GitHub (pinned to 806f444784)

Solutions

1. Bypass the intermediary by pointing the servers list directly at Elasticsearch/OpenSearch nodes and re-test.
2. Inspect the actual response body (tcpdump, proxy logs, or a snapshottest-style recorder) to find which middlebox alters it.
3. If a custom proxy is required, configure it to pass the _bulk response through unmodified.
4. Report/fix the compatibility layer so it returns one result item per document in order.

// validate that the configured endpoint is a real ES/OS node, not a
// custom gateway that may rewrite _bulk responses
resp, _ := http.Get(server + "/")
b, _ := io.ReadAll(resp.Body)
if !bytes.Contains(b, []byte("cluster_name")) {
    log.Fatalf("endpoint %s does not look like Elasticsearch", server)
}

// pre-flight shape check equivalent to the client's own invariant
func looksLikeBulkResponse(raw []byte, count int) bool {
    var r struct{ Items []json.RawMessage `json:"items"` }
    if json.Unmarshal(raw, &r) != nil { return false }
    return len(r.Items) == count
}

if err := writer.WriteBatch(ctx, items); err != nil {
    var malformed = strings.Contains(err.Error(), "malformed bulk response")
    if malformed {
        // response item count != sent docs: an intermediary is rewriting
        // bodies; fail loudly / bypass proxy rather than blind-retrying
    }
    return err
}

When it happens

Trigger: A proxy or intermediary returns a truncated or aggregated response; a custom gateway strips or merges the items array; a software/hardware load balancer returns its own partial body; essentially any middlebox that alters the _bulk response body while keeping it JSON-parseable.

Common situations: Corporate HTTP proxies that rewrite or buffer responses; misconfigured service meshes retrying and merging bodies; custom ES gateways (security or rate-limiting proxies) returning synthetic JSON; running against a mock/compatibility layer that does not implement _bulk faithfully.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to parse bulk response: %w
• bulk request failed: %w
• %d of %d bulk items rejected: %s
• file must begin with '['
• max spans count reached

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/2815d2a2ce0c8d39. Report an issue: GitHub.