jaegertracing/jaeger · critical

failed to send batch

Error message

failed to send batch: %w

What it means

After appending all span rows, WriteTraces flushes them with batch.Send, which makes clickhouse-go transmit the buffered batch and wait for the server's response. If the send or the server-side insert fails — connection drop, timeout, ClickHouse rejecting the block (too many parts, memory limit, deduplication/insert constraints) — the error is wrapped as "failed to send batch". All spans in the batch fail to persist even though appends succeeded locally.

Solutions

1. Check ClickHouse server logs for the insert rejection reason (memory limit, too many parts, read-only disk) and raise/resolve the server-side limit
2. Unwrap the error: context deadline exceeded means increase the write timeout or batch size is too large; dial errors mean connection/network problems
3. Retry WriteTraces — batch.Send failures are typically transient once the server recovers; consider enabling idempotency/dedup if duplicates are a concern
4. Tune batch sizing: aggregate fewer spans per batch if memory limits hit, or larger/more infrequent batches if 'Too many parts'
5. Verify ClickHouse disk space and replica health (system.replicas) in replicated deployments

Example fix

// before: huge batch blows server memory limit during Send
// Error: failed to send batch: code: 241, MEMORY_LIMIT_EXCEEDED
for spans := range hugeTraceStream {
    writer.WriteTraces(ctx, spans) // each call one giant batch
}

// after: cap the batch size before writing
const maxSpansPerBatch = 5000
for chunk := range chunked(hugeTraceStream, maxSpansPerBatch) {
    writer.WriteTraces(ctx, chunk)
}

if err := ctx.Err(); err != nil { return err }
if n := td.SpanCount(); n > maxSpansPerBatch { return fmt.Errorf("batch of %d exceeds limit %d", n, maxSpansPerBatch) }

err := retry.Do(func() error {
    return writer.WriteTraces(ctx, td) // retry Send failures with backoff
}, retry.Attempts(3), retry.BackOffDelay(time.Second, 2, 10*time.Second))
if err != nil && strings.Contains(err.Error(), "failed to send batch") {
    log.Error("clickhouse rejected batch after retries", "err", err)
}

When it happens

Trigger: Calling WriteTraces when: the connection dies between Append and Send, ctx deadline expires during flush, ClickHouse rejects the insert block due to server limits (max_memory_usage, 'Too many parts', max_insert_block_size), a replica is unavailable in a replicated setup, or a writeable table constraint rejects the data.

Common situations: Bulk-ingesting large trace batches that exceed ClickHouse memory limits; too-frequent small inserts causing 'Too many parts' (MergTree limit) — ironically fixed by bigger batches; network flakiness or LB idle timeout during a long flush; disk full on ClickHouse server.

Related errors

• failed to prepare batch
• failed to query trace IDs
• failed to append span to batch
• failed to query trace
• failed to read trace ID rows

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/ad933fe25ffba55a. Report an issue: GitHub.

Appendix: source

Thrown at internal/storage/v2/clickhouse/tracestore/writer.go:107

					sr.ScopeAttributes.BoolKeys,
					sr.ScopeAttributes.BoolValues,
					sr.ScopeAttributes.DoubleKeys,
					sr.ScopeAttributes.DoubleValues,
					sr.ScopeAttributes.IntKeys,
					sr.ScopeAttributes.IntValues,
					sr.ScopeAttributes.StrKeys,
					sr.ScopeAttributes.StrValues,
					sr.ScopeAttributes.ComplexKeys,
					sr.ScopeAttributes.ComplexValues,
				)
				if err != nil {
					return fmt.Errorf("failed to append span to batch: %w", err)
				}
			}
		}
	}
	if err := batch.Send(); err != nil {
		return fmt.Errorf("failed to send batch: %w", err)
	}
	return nil
}

func toTuple[T any](keys [][]string, values [][]T) [][][]any {
	tuple := make([][][]any, 0, len(keys))
	for i := range keys {
		inner := make([][]any, 0, len(keys[i]))
		for j := range keys[i] {
			inner = append(inner, []any{keys[i][j], values[i][j]})
		}
		tuple = append(tuple, inner)
	}
	return tuple
}

View on GitHub (pinned to 806f444784)