jaegertracing/jaeger · critical

failed to send batch: %w

Error message

failed to send batch: %w

What it means

After appending all span rows, WriteTraces flushes them with batch.Send, which makes clickhouse-go transmit the buffered batch and wait for the server's response. If the send or the server-side insert fails — connection drop, timeout, ClickHouse rejecting the block (too many parts, memory limit, deduplication/insert constraints) — the error is wrapped as "failed to send batch". All spans in the batch fail to persist even though appends succeeded locally.

Source

Thrown at internal/storage/v2/clickhouse/tracestore/writer.go:107

					sr.ScopeAttributes.BoolKeys,
					sr.ScopeAttributes.BoolValues,
					sr.ScopeAttributes.DoubleKeys,
					sr.ScopeAttributes.DoubleValues,
					sr.ScopeAttributes.IntKeys,
					sr.ScopeAttributes.IntValues,
					sr.ScopeAttributes.StrKeys,
					sr.ScopeAttributes.StrValues,
					sr.ScopeAttributes.ComplexKeys,
					sr.ScopeAttributes.ComplexValues,
				)
				if err != nil {
					return fmt.Errorf("failed to append span to batch: %w", err)
				}
			}
		}
	}
	if err := batch.Send(); err != nil {
		return fmt.Errorf("failed to send batch: %w", err)
	}
	return nil
}

func toTuple[T any](keys [][]string, values [][]T) [][][]any {
	tuple := make([][][]any, 0, len(keys))
	for i := range keys {
		inner := make([][]any, 0, len(keys[i]))
		for j := range keys[i] {
			inner = append(inner, []any{keys[i][j], values[i][j]})
		}
		tuple = append(tuple, inner)
	}
	return tuple
}

View on GitHub (pinned to 806f444784)

Solutions

1. Check ClickHouse server logs for the insert rejection reason (memory limit, too many parts, read-only disk) and raise/resolve the server-side limit
2. Unwrap the error: context deadline exceeded means increase the write timeout or batch size is too large; dial errors mean connection/network problems
3. Retry WriteTraces — batch.Send failures are typically transient once the server recovers; consider enabling idempotency/dedup if duplicates are a concern
4. Tune batch sizing: aggregate fewer spans per batch if memory limits hit, or larger/more infrequent batches if 'Too many parts'
5. Verify ClickHouse disk space and replica health (system.replicas) in replicated deployments

Example fix

// before: huge batch blows server memory limit during Send
// Error: failed to send batch: code: 241, MEMORY_LIMIT_EXCEEDED
for spans := range hugeTraceStream {
    writer.WriteTraces(ctx, spans) // each call one giant batch
}

// after: cap the batch size before writing
const maxSpansPerBatch = 5000
for chunk := range chunked(hugeTraceStream, maxSpansPerBatch) {
    writer.WriteTraces(ctx, chunk)
}

if err := ctx.Err(); err != nil { return err }
if n := td.SpanCount(); n > maxSpansPerBatch { return fmt.Errorf("batch of %d exceeds limit %d", n, maxSpansPerBatch) }

err := retry.Do(func() error {
    return writer.WriteTraces(ctx, td) // retry Send failures with backoff
}, retry.Attempts(3), retry.BackOffDelay(time.Second, 2, 10*time.Second))
if err != nil && strings.Contains(err.Error(), "failed to send batch") {
    log.Error("clickhouse rejected batch after retries", "err", err)
}

When it happens

Trigger: Calling WriteTraces when: the connection dies between Append and Send, ctx deadline expires during flush, ClickHouse rejects the insert block due to server limits (max_memory_usage, 'Too many parts', max_insert_block_size), a replica is unavailable in a replicated setup, or a writeable table constraint rejects the data.

Common situations: Bulk-ingesting large trace batches that exceed ClickHouse memory limits; too-frequent small inserts causing 'Too many parts' (MergTree limit) — ironically fixed by bigger batches; network flakiness or LB idle timeout during a long flush; disk full on ClickHouse server.

Related errors

• failed to query trace IDs: %w
• failed to prepare batch: %w
• failed to query trace: %w
• failed to read trace ID rows: %w
• failed to append span to batch: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/ad933fe25ffba55a. Report an issue: GitHub.