{"record":{"id":"a7de788c7343acf2","repo":"jaegertracing/jaeger","slug":"failed-to-append-span-to-batch-w","errorCode":null,"errorMessage":"failed to append span to batch: %w","messagePattern":"failed to append span to batch: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/v2/clickhouse/tracestore/writer.go","lineNumber":101,"sourceCode":"\t\t\t\t\tsr.ResourceAttributes.StrKeys,\n\t\t\t\t\tsr.ResourceAttributes.StrValues,\n\t\t\t\t\tsr.ResourceAttributes.ComplexKeys,\n\t\t\t\t\tsr.ResourceAttributes.ComplexValues,\n\t\t\t\t\tsr.ScopeName,\n\t\t\t\t\tsr.ScopeVersion,\n\t\t\t\t\tsr.ScopeAttributes.BoolKeys,\n\t\t\t\t\tsr.ScopeAttributes.BoolValues,\n\t\t\t\t\tsr.ScopeAttributes.DoubleKeys,\n\t\t\t\t\tsr.ScopeAttributes.DoubleValues,\n\t\t\t\t\tsr.ScopeAttributes.IntKeys,\n\t\t\t\t\tsr.ScopeAttributes.IntValues,\n\t\t\t\t\tsr.ScopeAttributes.StrKeys,\n\t\t\t\t\tsr.ScopeAttributes.StrValues,\n\t\t\t\t\tsr.ScopeAttributes.ComplexKeys,\n\t\t\t\t\tsr.ScopeAttributes.ComplexValues,\n\t\t\t\t)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to append span to batch: %w\", err)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\tif err := batch.Send(); err != nil {\n\t\treturn fmt.Errorf(\"failed to send batch: %w\", err)\n\t}\n\treturn nil\n}\n\nfunc toTuple[T any](keys [][]string, values [][]T) [][][]any {\n\ttuple := make([][][]any, 0, len(keys))\n\tfor i := range keys {\n\t\tinner := make([][]any, 0, len(keys[i]))\n\t\tfor j := range keys[i] {\n\t\t\tinner = append(inner, []any{keys[i][j], values[i][j]})\n\t\t}\n\t\ttuple = append(tuple, inner)","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v2/clickhouse/tracestore/writer.go#L83-L119","documentation":"For each span, WriteTraces appends a row with ~50 ordered column values to the prepared batch via batch.Append. If the driver rejects the append — wrong number of arguments vs the INSERT column list, Go value type not convertible to the ClickHouse column type (e.g. tuple/array shape mismatch), or NULL in a non-nullable column — the loop aborts with \"failed to append span to batch\". Rows already appended are discarded because Send never happens (the deferred batch.Close releases them).","triggerScenarios":"Calling WriteTraces with span data that produces a row not matching sql.InsertSpan's schema: attribute arrays whose parallel key/value slices differ in length, event/link tuples in unexpected shape, empty required fields (e.g. empty service name in a non-nullable column), or a Jaeger/dbmodel version mismatch with the table schema so Append receives too few/many values.","commonSituations":"OTLP data with unusual attribute types hitting a dbmodel conversion gap; schema upgrade applied to ClickHouse but Jaeger writer not upgraded (or vice versa) changing the column count; corrupted span data from a malformed collector payload; ClickHouse Map/Array column type changed to non-nullable.","solutions":["Read the wrapped driver error — clickhouse-go reports the column index and expected/actual type","Confirm the ClickHouse spans table schema matches the sql.InsertSpan column list for this Jaeger version; re-run migrations if they drifted","Validate span data before writing: consistent attribute key/value array lengths, non-empty service name, valid trace/span IDs","Check for a version skew between the Jaeger writer and the deployed schema and align them","Log the offending span (trace ID, span ID) to identify the producing telemetry source and fix it upstream"],"exampleFix":"// before: attribute conversion yields mismatched key/value arrays\nattrs.BoolKeys = [\"k1\", \"k2\"]; attrs.BoolValues = [true] // length mismatch -> Append fails\n\n// after: keep parallel arrays consistent in dbmodel attribute conversion\nif len(keys) != len(values) { /* pad or drop orphans consistently */ }","handlingStrategy":"validation","validationCode":"func validateRow(sr dbmodel.Row) error {\n    if len(sr.TraceID) == 0 || len(sr.ID) == 0 {\n        return errors.New(\"span missing trace/span ID\")\n    }\n    if sr.ServiceName == \"\" {\n        return errors.New(\"span missing service name\")\n    }\n    attrs := []struct{ k, v [][]string }{\n        {sr.Attributes.BoolKeys, nil}, // check all key/value pairs for equal lengths\n    }\n    _ = attrs\n    return nil\n}\nif err := validateRow(sr); err != nil { skipSpan(err) }","typeGuard":"func parallelKV[T any](keys []string, values []T) bool { return len(keys) == len(values) }","tryCatchPattern":"if err := writer.WriteTraces(ctx, td); err != nil {\n    if strings.Contains(err.Error(), \"failed to append span to batch\") {\n        // find offending span: bisect by re-writing spans individually\n        log.Error(\"span rejected by batch append; quarantining batch\", \"err\", err)\n        quarantine(td)\n        return err\n    }\n    return err\n}","preventionTips":["Unit-test dbmodel.ToRow against edge-case OTLP payloads (empty attrs, unusual types)","Keep attribute key/value arrays strictly parallel during conversion","Keep Jaeger writer and ClickHouse schema on the same version","Quarantine and log bad batches instead of blocking the whole pipeline"],"tags":["clickhouse","go","batch-insert","data-integrity","schema-mismatch"],"backgroundTag":"column-type-mismatch","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}