{"record":{"id":"2815d2a2ce0c8d39","repo":"jaegertracing/jaeger","slug":"malformed-bulk-response-d-item-results-for-d-do","errorCode":null,"errorMessage":"malformed bulk response: %d item results for %d documents","messagePattern":"malformed bulk response: (.+?) item results for (.+?) documents","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/elasticsearch/esclient/sync_bulk.go","lineNumber":189,"sourceCode":"\t\tendpoint:    \"_bulk\",\n\t\tmethod:      http.MethodPost,\n\t\tbody:        body,\n\t\tcontentType: \"application/x-ndjson\",\n\t})\n\tif err != nil {\n\t\treturn 0, fmt.Errorf(\"bulk request failed: %w\", err)\n\t}\n\n\tvar resp bulkResponse\n\tif err := json.Unmarshal(raw, &resp); err != nil {\n\t\treturn 0, fmt.Errorf(\"failed to parse bulk response: %w\", err)\n\t}\n\t// A well-formed _bulk response reports exactly one result per document, in\n\t// request order. If a proxy or partial response returns fewer (or more), the\n\t// per-item accounting below can't be trusted, so fail the whole chunk (0\n\t// durable) rather than silently miscount — the caller retries the batch.\n\tif len(resp.Items) != count {\n\t\treturn 0, fmt.Errorf(\"malformed bulk response: %d item results for %d documents\", len(resp.Items), count)\n\t}\n\t// The HTTP round-trip and response parsing succeeded, so record latency-ok even\n\t// when some items are rejected below. This matches the async BulkIndexer, whose\n\t// latency-err covers only a whole-request (transport/non-2xx) failure while\n\t// per-item rejections are reflected through the errors counter alone; latency-err\n\t// therefore keeps its meaning of \"the request failed\", not \"some items failed\".\n\tsuccess = true\n\t// Derive failures from the per-item statuses, not the top-level `errors` flag:\n\t// a malformed or proxied response could report errors:false while an item still\n\t// carries a failing status, and silently succeeding there would advance the\n\t// Kafka offset over lost data — exactly what this synchronous writer prevents.\n\tout := resp.classify()\n\tif out.conflicts > 0 {\n\t\t// Counted in the durable total returned below, mirroring the async indexer's\n\t\t// onItemConflict: an already-present document achieved the write's goal.\n\t\tw.logger.Debug(\"bulk items already present (idempotent write)\",\n\t\t\tzap.Int(\"conflicts\", out.conflicts), zap.Int(\"total\", count),\n\t\t\tzap.Int(\"status\", http.StatusConflict))","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/elasticsearch/esclient/sync_bulk.go#L171-L207","documentation":"A well-formed _bulk response contains exactly one result per submitted document, in request order. When the parsed response has a different number of item entries than documents sent, the per-item accounting cannot be trusted, so sendChunk fails the entire chunk (0 durable) rather than silently miscounting — the caller retries the batch.","triggerScenarios":"A proxy or intermediary returns a truncated or aggregated response; a custom gateway strips or merges the items array; a software/hardware load balancer returns its own partial body; essentially any middlebox that alters the _bulk response body while keeping it JSON-parseable.","commonSituations":"Corporate HTTP proxies that rewrite or buffer responses; misconfigured service meshes retrying and merging bodies; custom ES gateways (security or rate-limiting proxies) returning synthetic JSON; running against a mock/compatibility layer that does not implement _bulk faithfully.","solutions":["Bypass the intermediary by pointing the servers list directly at Elasticsearch/OpenSearch nodes and re-test.","Inspect the actual response body (tcpdump, proxy logs, or a snapshottest-style recorder) to find which middlebox alters it.","If a custom proxy is required, configure it to pass the _bulk response through unmodified.","Report/fix the compatibility layer so it returns one result item per document in order."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// validate that the configured endpoint is a real ES/OS node, not a\n// custom gateway that may rewrite _bulk responses\nresp, _ := http.Get(server + \"/\")\nb, _ := io.ReadAll(resp.Body)\nif !bytes.Contains(b, []byte(\"cluster_name\")) {\n    log.Fatalf(\"endpoint %s does not look like Elasticsearch\", server)\n}","typeGuard":"// pre-flight shape check equivalent to the client's own invariant\nfunc looksLikeBulkResponse(raw []byte, count int) bool {\n    var r struct{ Items []json.RawMessage `json:\"items\"` }\n    if json.Unmarshal(raw, &r) != nil { return false }\n    return len(r.Items) == count\n}","tryCatchPattern":"if err := writer.WriteBatch(ctx, items); err != nil {\n    var malformed = strings.Contains(err.Error(), \"malformed bulk response\")\n    if malformed {\n        // response item count != sent docs: an intermediary is rewriting\n        // bodies; fail loudly / bypass proxy rather than blind-retrying\n    }\n    return err\n}","preventionTips":["Do not place body-rewriting proxies or caching layers in the _bulk path.","Test with a snapshot/recorded server harness when introducing new gateway components.","Ensure any security/rate-limit proxy forwards the upstream body byte-for-byte.","Keep cluster networking direct (client -> ES node) where possible."],"tags":["elasticsearch","bulk-index","proxy","protocol-violation"],"backgroundTag":"malformed-bulk-response","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}