jaegertracing/jaeger · error

failed to append span to batch: %w

Error message

failed to append span to batch: %w

What it means

For each span, WriteTraces appends a row with ~50 ordered column values to the prepared batch via batch.Append. If the driver rejects the append — wrong number of arguments vs the INSERT column list, Go value type not convertible to the ClickHouse column type (e.g. tuple/array shape mismatch), or NULL in a non-nullable column — the loop aborts with "failed to append span to batch". Rows already appended are discarded because Send never happens (the deferred batch.Close releases them).

Source

Thrown at internal/storage/v2/clickhouse/tracestore/writer.go:101

					sr.ResourceAttributes.StrKeys,
					sr.ResourceAttributes.StrValues,
					sr.ResourceAttributes.ComplexKeys,
					sr.ResourceAttributes.ComplexValues,
					sr.ScopeName,
					sr.ScopeVersion,
					sr.ScopeAttributes.BoolKeys,
					sr.ScopeAttributes.BoolValues,
					sr.ScopeAttributes.DoubleKeys,
					sr.ScopeAttributes.DoubleValues,
					sr.ScopeAttributes.IntKeys,
					sr.ScopeAttributes.IntValues,
					sr.ScopeAttributes.StrKeys,
					sr.ScopeAttributes.StrValues,
					sr.ScopeAttributes.ComplexKeys,
					sr.ScopeAttributes.ComplexValues,
				)
				if err != nil {
					return fmt.Errorf("failed to append span to batch: %w", err)
				}
			}
		}
	}
	if err := batch.Send(); err != nil {
		return fmt.Errorf("failed to send batch: %w", err)
	}
	return nil
}

func toTuple[T any](keys [][]string, values [][]T) [][][]any {
	tuple := make([][][]any, 0, len(keys))
	for i := range keys {
		inner := make([][]any, 0, len(keys[i]))
		for j := range keys[i] {
			inner = append(inner, []any{keys[i][j], values[i][j]})
		}
		tuple = append(tuple, inner)

View on GitHub (pinned to 806f444784)

Solutions

1. Read the wrapped driver error — clickhouse-go reports the column index and expected/actual type
2. Confirm the ClickHouse spans table schema matches the sql.InsertSpan column list for this Jaeger version; re-run migrations if they drifted
3. Validate span data before writing: consistent attribute key/value array lengths, non-empty service name, valid trace/span IDs
4. Check for a version skew between the Jaeger writer and the deployed schema and align them
5. Log the offending span (trace ID, span ID) to identify the producing telemetry source and fix it upstream

Example fix

// before: attribute conversion yields mismatched key/value arrays
attrs.BoolKeys = ["k1", "k2"]; attrs.BoolValues = [true] // length mismatch -> Append fails

// after: keep parallel arrays consistent in dbmodel attribute conversion
if len(keys) != len(values) { /* pad or drop orphans consistently */ }

func validateRow(sr dbmodel.Row) error {
    if len(sr.TraceID) == 0 || len(sr.ID) == 0 {
        return errors.New("span missing trace/span ID")
    }
    if sr.ServiceName == "" {
        return errors.New("span missing service name")
    }
    attrs := []struct{ k, v [][]string }{
        {sr.Attributes.BoolKeys, nil}, // check all key/value pairs for equal lengths
    }
    _ = attrs
    return nil
}
if err := validateRow(sr); err != nil { skipSpan(err) }

func parallelKV[T any](keys []string, values []T) bool { return len(keys) == len(values) }

if err := writer.WriteTraces(ctx, td); err != nil {
    if strings.Contains(err.Error(), "failed to append span to batch") {
        // find offending span: bisect by re-writing spans individually
        log.Error("span rejected by batch append; quarantining batch", "err", err)
        quarantine(td)
        return err
    }
    return err
}

When it happens

Trigger: Calling WriteTraces with span data that produces a row not matching sql.InsertSpan's schema: attribute arrays whose parallel key/value slices differ in length, event/link tuples in unexpected shape, empty required fields (e.g. empty service name in a non-nullable column), or a Jaeger/dbmodel version mismatch with the table schema so Append receives too few/many values.

Common situations: OTLP data with unusual attribute types hitting a dbmodel conversion gap; schema upgrade applied to ClickHouse but Jaeger writer not upgraded (or vice versa) changing the column count; corrupted span data from a malformed collector payload; ClickHouse Map/Array column type changed to non-nullable.

Related errors

• failed to prepare batch: %w
• failed to decode trace ID: %w
• failed to send batch: %w
• failed to query trace IDs: %w
• failed to read trace ID rows: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/a7de788c7343acf2. Report an issue: GitHub.