{"record":{"id":"874a940a147c6070","repo":"jaegertracing/jaeger","slug":"failed-to-parse-bulk-response-w","errorCode":null,"errorMessage":"failed to parse bulk response: %w","messagePattern":"failed to parse bulk response: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/elasticsearch/esclient/sync_bulk.go","lineNumber":182,"sourceCode":"\t\t\tw.metrics.LatencyOk.Record(time.Since(start))\n\t\t} else {\n\t\t\tw.metrics.LatencyErr.Record(time.Since(start))\n\t\t}\n\t}()\n\n\traw, err := w.client.request(ctx, elasticRequest{\n\t\tendpoint:    \"_bulk\",\n\t\tmethod:      http.MethodPost,\n\t\tbody:        body,\n\t\tcontentType: \"application/x-ndjson\",\n\t})\n\tif err != nil {\n\t\treturn 0, fmt.Errorf(\"bulk request failed: %w\", err)\n\t}\n\n\tvar resp bulkResponse\n\tif err := json.Unmarshal(raw, &resp); err != nil {\n\t\treturn 0, fmt.Errorf(\"failed to parse bulk response: %w\", err)\n\t}\n\t// A well-formed _bulk response reports exactly one result per document, in\n\t// request order. If a proxy or partial response returns fewer (or more), the\n\t// per-item accounting below can't be trusted, so fail the whole chunk (0\n\t// durable) rather than silently miscount — the caller retries the batch.\n\tif len(resp.Items) != count {\n\t\treturn 0, fmt.Errorf(\"malformed bulk response: %d item results for %d documents\", len(resp.Items), count)\n\t}\n\t// The HTTP round-trip and response parsing succeeded, so record latency-ok even\n\t// when some items are rejected below. This matches the async BulkIndexer, whose\n\t// latency-err covers only a whole-request (transport/non-2xx) failure while\n\t// per-item rejections are reflected through the errors counter alone; latency-err\n\t// therefore keeps its meaning of \"the request failed\", not \"some items failed\".\n\tsuccess = true\n\t// Derive failures from the per-item statuses, not the top-level `errors` flag:\n\t// a malformed or proxied response could report errors:false while an item still\n\t// carries a failing status, and silently succeeding there would advance the\n\t// Kafka offset over lost data — exactly what this synchronous writer prevents.","sourceCodeStart":164,"sourceCodeEnd":200,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/elasticsearch/esclient/sync_bulk.go#L164-L200","documentation":"After a successful HTTP round-trip, sendChunk JSON-decodes the _bulk response body into bulkResponse. This error means the body the client received was not the expected JSON shape ({\"items\":[...]}), so per-item durability cannot be determined. The chunk is reported as 0 durable and the caller retries.","triggerScenarios":"The client.request call returns raw bytes that json.Unmarshal cannot decode into bulkResponse — e.g. an HTML error page from a proxy, a truncated body, a gzip/encoding mismatch, or an auth gateway returning non-JSON on a 200-level or odd response.","commonSituations":"A reverse proxy or load balancer (nginx, Istio, AWS ALB) intercepting the request and returning an HTML 502/401 page; a captive portal; a broken intermediary stripping Content-Encoding; hitting a non-Elasticsearch endpoint at the configured URL.","solutions":["Curl the configured server URL for /_bulk or / to see what is actually answering; fix the URL or remove the intercepting proxy.","Ensure no proxy rewrites/intercepts the response body; check proxy error pages in the path.","Verify the endpoint answers with ES/OS JSON: curl -s http://host:9200 | head and confirm the version JSON.","Check for TLS-terminating layers returning plaintext HTML errors and correct certs/ports."],"exampleFix":"// before: URL points at a proxy UI that returns HTML\nservers: [\"http://proxy.internal/kibana\"]\n// after: point at the ES data node/API\nservers: [\"http://es-client-http.logging:9200\"]","handlingStrategy":"type-guard","validationCode":"// confirm the endpoint speaks Elasticsearch JSON before sending data\nresp, err := http.Get(server + \"/\")\nif err == nil {\n    var v map[string]any\n    if json.NewDecoder(resp.Body).Decode(&v) != nil || v[\"cluster_name\"] == nil {\n        log.Fatalf(\"%s is not an Elasticsearch/OpenSearch endpoint\", server)\n    }\n    resp.Body.Close()\n}","typeGuard":"// probe the raw response shape before trusting per-item data\nvar probe struct {\n    Items []json.RawMessage `json:\"items\"`\n}\nif json.Unmarshal(raw, &probe) != nil || len(probe.Items) == 0 {\n    // response body is not an ES bulk response — treat as transport/intermediary failure\n}","tryCatchPattern":"err := writer.WriteBatch(ctx, items)\nif err != nil && strings.Contains(err.Error(), \"failed to parse bulk response\") {\n    // body wasn't ES JSON: check proxies/gateways before retrying\n    log.Errorf(\"non-JSON response from ES path: %v\", err)\n}","preventionTips":["Point the servers list directly at ES/OpenSearch API nodes, not UI or proxy pages.","Bypass or reconfigure middleboxes that inject HTML error pages (nginx, Istio, ALB).","Verify with curl that the URL returns Elasticsearch's version JSON before deploying.","Watch for Content-Encoding mismatches when compression is enabled in the client config."],"tags":["elasticsearch","json","bulk-index","proxy"],"backgroundTag":"invalid-json-response","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}