{"record":{"id":"1925d3c9a8068b80","repo":"jaegertracing/jaeger","slug":"bulk-request-failed-w","errorCode":null,"errorMessage":"bulk request failed: %w","messagePattern":"bulk request failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"internal/storage/elasticsearch/esclient/sync_bulk.go","lineNumber":177,"sourceCode":"func (w *SyncBulkWriter) sendChunk(ctx context.Context, body []byte, count int) (int, error) {\n\tstart := time.Now()\n\tsuccess := false\n\tdefer func() {\n\t\tif success {\n\t\t\tw.metrics.LatencyOk.Record(time.Since(start))\n\t\t} else {\n\t\t\tw.metrics.LatencyErr.Record(time.Since(start))\n\t\t}\n\t}()\n\n\traw, err := w.client.request(ctx, elasticRequest{\n\t\tendpoint:    \"_bulk\",\n\t\tmethod:      http.MethodPost,\n\t\tbody:        body,\n\t\tcontentType: \"application/x-ndjson\",\n\t})\n\tif err != nil {\n\t\treturn 0, fmt.Errorf(\"bulk request failed: %w\", err)\n\t}\n\n\tvar resp bulkResponse\n\tif err := json.Unmarshal(raw, &resp); err != nil {\n\t\treturn 0, fmt.Errorf(\"failed to parse bulk response: %w\", err)\n\t}\n\t// A well-formed _bulk response reports exactly one result per document, in\n\t// request order. If a proxy or partial response returns fewer (or more), the\n\t// per-item accounting below can't be trusted, so fail the whole chunk (0\n\t// durable) rather than silently miscount — the caller retries the batch.\n\tif len(resp.Items) != count {\n\t\treturn 0, fmt.Errorf(\"malformed bulk response: %d item results for %d documents\", len(resp.Items), count)\n\t}\n\t// The HTTP round-trip and response parsing succeeded, so record latency-ok even\n\t// when some items are rejected below. This matches the async BulkIndexer, whose\n\t// latency-err covers only a whole-request (transport/non-2xx) failure while\n\t// per-item rejections are reflected through the errors counter alone; latency-err\n\t// therefore keeps its meaning of \"the request failed\", not \"some items failed\".","sourceCodeStart":159,"sourceCodeEnd":195,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/elasticsearch/esclient/sync_bulk.go#L159-L195","documentation":"SyncBulkWriter.sendChunk POSTs an NDJSON chunk to the Elasticsearch/OpenSearch _bulk endpoint. This error wraps any transport-level failure of that HTTP request — connection refused, DNS failure, TLS handshake error, non-2xx status, or a canceled/expired context. The whole chunk is reported as 0 durable documents so the caller can retry the entire batch.","triggerScenarios":"Calling SyncBulkWriter.WriteBatch when the ES/OS node is unreachable (connection refused/DNS), the context is canceled or its deadline expires mid-request, the base RoundTripper fails (TLS, auth proxy), or the server returns a non-2xx HTTP status (e.g. 413 for an oversized body, 503 during a cluster outage).","commonSituations":"Elasticsearch pod down or restarting during a Kubernetes rolling update; wrong host/port in the servers config; http.max_content_length exceeded by a chunk or single huge span document (413); load balancer returning 502/503; network partition between Jaeger and the storage backend; context deadline too short for large batches.","solutions":["Verify the Elasticsearch/OpenSearch servers list points at reachable nodes: curl the URL from the Jaeger host (e.g. curl http://host:9200) and fix addresses/ports.","If the error wraps a 413, lower the bulk maxBytes config (or ES http.max_content_length) and split oversized spans.","If the error wraps 429/5xx, reduce batch size or increase exporter retry/backoff so the cluster can absorb the load.","If it wraps 'context deadline exceeded', increase the request/exporter timeout or network limits.","Check TLS/credentials: the failure can come from the base RoundTripper stack (certs, API key, SigV4)."],"exampleFix":"// before: chunk too large -> 413 from ES\nmaxBytes := 200 * 1024 * 1024\n// after: stay well under http.max_content_length (default 100 MB)\nmaxBytes := 5 * 1024 * 1024","handlingStrategy":"retry","validationCode":"// before wiring the client, verify each server answers\nfor _, s := range servers {\n    resp, err := http.Get(s + \"/\")\n    if err != nil { log.Fatalf(\"ES unreachable %s: %v\", s, err) }\n    resp.Body.Close()\n}","typeGuard":"// unwrap and inspect the underlying cause\nvar netErr net.Error\nif errors.As(err, &netErr) && netErr.Timeout() {\n    // retry with backoff\n}\nif errors.Is(err, context.DeadlineExceeded) { /* increase timeout */ }","tryCatchPattern":"err := writer.WriteBatch(ctx, items)\nif err != nil {\n    if strings.Contains(err.Error(), \"bulk request failed\") {\n        // transport-level: whole chunk lost, safe to retry everything\n        retryWithBackoff(ctx, items)\n    }\n    return fmt.Errorf(\"span export failed: %w\", err)\n}","preventionTips":["Health-check every configured server at startup before accepting traffic.","Keep chunk maxBytes well under the backend's http.max_content_length (default 100 MB).","Set exporter timeouts generously relative to batch size; large batches need seconds, not milliseconds.","Use ready/liveness probes so Jaeger instances drain during ES rolling restarts.","Rely on the exporter's retry queue: transport failures lose the whole chunk, so at-least-once redelivery must be in place."],"tags":["elasticsearch","network","bulk-index","retryable"],"backgroundTag":"bulk-request-transport-failed","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}