What causes "bulk request failed: %w" in jaegertracing/jaeger?

Verify the Elasticsearch/OpenSearch servers list points at reachable nodes: curl the URL from the Jaeger host (e.g. curl http://host:9200) and fix addresses/ports.

jaegertracing/jaeger · critical

bulk request failed: %w

Error message

bulk request failed: %w

What it means

SyncBulkWriter.sendChunk POSTs an NDJSON chunk to the Elasticsearch/OpenSearch _bulk endpoint. This error wraps any transport-level failure of that HTTP request — connection refused, DNS failure, TLS handshake error, non-2xx status, or a canceled/expired context. The whole chunk is reported as 0 durable documents so the caller can retry the entire batch.

Source

Thrown at internal/storage/elasticsearch/esclient/sync_bulk.go:177

func (w *SyncBulkWriter) sendChunk(ctx context.Context, body []byte, count int) (int, error) {
	start := time.Now()
	success := false
	defer func() {
		if success {
			w.metrics.LatencyOk.Record(time.Since(start))
		} else {
			w.metrics.LatencyErr.Record(time.Since(start))
		}
	}()

	raw, err := w.client.request(ctx, elasticRequest{
		endpoint:    "_bulk",
		method:      http.MethodPost,
		body:        body,
		contentType: "application/x-ndjson",
	})
	if err != nil {
		return 0, fmt.Errorf("bulk request failed: %w", err)
	}

	var resp bulkResponse
	if err := json.Unmarshal(raw, &resp); err != nil {
		return 0, fmt.Errorf("failed to parse bulk response: %w", err)
	}
	// A well-formed _bulk response reports exactly one result per document, in
	// request order. If a proxy or partial response returns fewer (or more), the
	// per-item accounting below can't be trusted, so fail the whole chunk (0
	// durable) rather than silently miscount — the caller retries the batch.
	if len(resp.Items) != count {
		return 0, fmt.Errorf("malformed bulk response: %d item results for %d documents", len(resp.Items), count)
	}
	// The HTTP round-trip and response parsing succeeded, so record latency-ok even
	// when some items are rejected below. This matches the async BulkIndexer, whose
	// latency-err covers only a whole-request (transport/non-2xx) failure while
	// per-item rejections are reflected through the errors counter alone; latency-err
	// therefore keeps its meaning of "the request failed", not "some items failed".

View on GitHub (pinned to 806f444784)

Solutions

Verify the Elasticsearch/OpenSearch servers list points at reachable nodes: curl the URL from the Jaeger host (e.g. curl http://host:9200) and fix addresses/ports.
If the error wraps a 413, lower the bulk maxBytes config (or ES http.max_content_length) and split oversized spans.
If the error wraps 429/5xx, reduce batch size or increase exporter retry/backoff so the cluster can absorb the load.
If it wraps 'context deadline exceeded', increase the request/exporter timeout or network limits.
Check TLS/credentials: the failure can come from the base RoundTripper stack (certs, API key, SigV4).

Example fix

// before: chunk too large -> 413 from ES
maxBytes := 200 * 1024 * 1024
// after: stay well under http.max_content_length (default 100 MB)
maxBytes := 5 * 1024 * 1024

Defensive patterns

Strategy: retry

Validate before calling

// before wiring the client, verify each server answers
for _, s := range servers {
    resp, err := http.Get(s + "/")
    if err != nil { log.Fatalf("ES unreachable %s: %v", s, err) }
    resp.Body.Close()
}

Type guard

// unwrap and inspect the underlying cause
var netErr net.Error
if errors.As(err, &netErr) && netErr.Timeout() {
    // retry with backoff
}
if errors.Is(err, context.DeadlineExceeded) { /* increase timeout */ }

Try / catch

err := writer.WriteBatch(ctx, items)
if err != nil {
    if strings.Contains(err.Error(), "bulk request failed") {
        // transport-level: whole chunk lost, safe to retry everything
        retryWithBackoff(ctx, items)
    }
    return fmt.Errorf("span export failed: %w", err)
}

Prevention

Health-check every configured server at startup before accepting traffic.
Keep chunk maxBytes well under the backend's http.max_content_length (default 100 MB).
Set exporter timeouts generously relative to batch size; large batches need seconds, not milliseconds.
Use ready/liveness probes so Jaeger instances drain during ES rolling restarts.
Rely on the exporter's retry queue: transport failures lose the whole chunk, so at-least-once redelivery must be in place.

When it happens

Trigger: Calling SyncBulkWriter.WriteBatch when the ES/OS node is unreachable (connection refused/DNS), the context is canceled or its deadline expires mid-request, the base RoundTripper fails (TLS, auth proxy), or the server returns a non-2xx HTTP status (e.g. 413 for an oversized body, 503 during a cluster outage).

Common situations: Elasticsearch pod down or restarting during a Kubernetes rolling update; wrong host/port in the servers config; http.max_content_length exceeded by a chunk or single huge span document (413); load balancer returning 502/503; network partition between Jaeger and the storage backend; context deadline too short for large batches.

Related errors

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/1925d3c9a8068b80. Report an issue: GitHub.