{"record":{"id":"ad933fe25ffba55a","repo":"jaegertracing/jaeger","slug":"failed-to-send-batch-w-ad933f","errorCode":null,"errorMessage":"failed to send batch: %w","messagePattern":"failed to send batch: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"internal/storage/v2/clickhouse/tracestore/writer.go","lineNumber":107,"sourceCode":"\t\t\t\t\tsr.ScopeAttributes.BoolKeys,\n\t\t\t\t\tsr.ScopeAttributes.BoolValues,\n\t\t\t\t\tsr.ScopeAttributes.DoubleKeys,\n\t\t\t\t\tsr.ScopeAttributes.DoubleValues,\n\t\t\t\t\tsr.ScopeAttributes.IntKeys,\n\t\t\t\t\tsr.ScopeAttributes.IntValues,\n\t\t\t\t\tsr.ScopeAttributes.StrKeys,\n\t\t\t\t\tsr.ScopeAttributes.StrValues,\n\t\t\t\t\tsr.ScopeAttributes.ComplexKeys,\n\t\t\t\t\tsr.ScopeAttributes.ComplexValues,\n\t\t\t\t)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to append span to batch: %w\", err)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\tif err := batch.Send(); err != nil {\n\t\treturn fmt.Errorf(\"failed to send batch: %w\", err)\n\t}\n\treturn nil\n}\n\nfunc toTuple[T any](keys [][]string, values [][]T) [][][]any {\n\ttuple := make([][][]any, 0, len(keys))\n\tfor i := range keys {\n\t\tinner := make([][]any, 0, len(keys[i]))\n\t\tfor j := range keys[i] {\n\t\t\tinner = append(inner, []any{keys[i][j], values[i][j]})\n\t\t}\n\t\ttuple = append(tuple, inner)\n\t}\n\treturn tuple\n}\n","sourceCodeStart":89,"sourceCodeEnd":123,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v2/clickhouse/tracestore/writer.go#L89-L123","documentation":"After appending all span rows, WriteTraces flushes them with batch.Send, which makes clickhouse-go transmit the buffered batch and wait for the server's response. If the send or the server-side insert fails — connection drop, timeout, ClickHouse rejecting the block (too many parts, memory limit, deduplication/insert constraints) — the error is wrapped as \"failed to send batch\". All spans in the batch fail to persist even though appends succeeded locally.","triggerScenarios":"Calling WriteTraces when: the connection dies between Append and Send, ctx deadline expires during flush, ClickHouse rejects the insert block due to server limits (max_memory_usage, 'Too many parts', max_insert_block_size), a replica is unavailable in a replicated setup, or a writeable table constraint rejects the data.","commonSituations":"Bulk-ingesting large trace batches that exceed ClickHouse memory limits; too-frequent small inserts causing 'Too many parts' (MergTree limit) — ironically fixed by bigger batches; network flakiness or LB idle timeout during a long flush; disk full on ClickHouse server.","solutions":["Check ClickHouse server logs for the insert rejection reason (memory limit, too many parts, read-only disk) and raise/resolve the server-side limit","Unwrap the error: context deadline exceeded means increase the write timeout or batch size is too large; dial errors mean connection/network problems","Retry WriteTraces — batch.Send failures are typically transient once the server recovers; consider enabling idempotency/dedup if duplicates are a concern","Tune batch sizing: aggregate fewer spans per batch if memory limits hit, or larger/more infrequent batches if 'Too many parts'","Verify ClickHouse disk space and replica health (system.replicas) in replicated deployments"],"exampleFix":"// before: huge batch blows server memory limit during Send\n// Error: failed to send batch: code: 241, MEMORY_LIMIT_EXCEEDED\nfor spans := range hugeTraceStream {\n    writer.WriteTraces(ctx, spans) // each call one giant batch\n}\n\n// after: cap the batch size before writing\nconst maxSpansPerBatch = 5000\nfor chunk := range chunked(hugeTraceStream, maxSpansPerBatch) {\n    writer.WriteTraces(ctx, chunk)\n}","handlingStrategy":"retry","validationCode":"if err := ctx.Err(); err != nil { return err }\nif n := td.SpanCount(); n > maxSpansPerBatch { return fmt.Errorf(\"batch of %d exceeds limit %d\", n, maxSpansPerBatch) }","typeGuard":null,"tryCatchPattern":"err := retry.Do(func() error {\n    return writer.WriteTraces(ctx, td) // retry Send failures with backoff\n}, retry.Attempts(3), retry.BackOffDelay(time.Second, 2, 10*time.Second))\nif err != nil && strings.Contains(err.Error(), \"failed to send batch\") {\n    log.Error(\"clickhouse rejected batch after retries\", \"err\", err)\n}","preventionTips":["Size batches to stay below ClickHouse max_insert_block_size and memory limits","Avoid high-frequency tiny inserts that trigger 'Too many parts'","Set write context deadlines comfortably above batch flush time","Monitor ClickHouse disk space and replica health","Enable insert deduplication (replicated tables) so retries are idempotent"],"tags":["clickhouse","go","batch-insert","database","network"],"backgroundTag":"clickhouse-insert-failed","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}