jaegertracing/jaeger · error

failed to decode trace ID: %w

Error message

failed to decode trace ID: %w

What it means

After scanning the trace ID as a hex string, readRowIntoTraceID decodes it with hex.DecodeString and converts the bytes to a pcommon.TraceID (16 bytes). If the string is not valid hex or its length is not 32 hex chars (16 bytes), the error is wrapped as "failed to decode trace ID". pcommon.TraceID requires exactly 16 bytes; shorter or longer inputs yield a garbage or invalid ID.

Source

Thrown at internal/storage/v2/clickhouse/tracestore/reader.go:234

			errs = append(errs, fmt.Errorf("failed to close rows: %w", closeErr))
		}
		if err := errors.Join(errs...); err != nil {
			yield(nil, err)
		}
	}
}

func readRowIntoTraceID(rows driver.Rows) ([]tracestore.FoundTraceID, error) {
	var traceIDHex string
	var start, end time.Time

	if err := rows.Scan(&traceIDHex, &start, &end); err != nil {
		return nil, fmt.Errorf("failed to scan row: %w", err)
	}

	b, err := hex.DecodeString(traceIDHex)
	if err != nil {
		return nil, fmt.Errorf("failed to decode trace ID: %w", err)
	}

	traceID := tracestore.FoundTraceID{
		TraceID: pcommon.TraceID(b),
	}

	if !start.IsZero() {
		traceID.Start = start
	}
	if !end.IsZero() {
		traceID.End = end
	}

	return []tracestore.FoundTraceID{
		traceID,
	}, nil
}

View on GitHub (pinned to 806f444784)

Solutions

1. Inspect the offending rows: SELECT trace_id FROM traces WHERE NOT match(trace_id, '^[0-9a-fA-F]{32}$') to find malformed IDs
2. Ensure all writers into the table encode trace IDs as 32-char lowercase hex (UTF8BytesString of the 16-byte ID)
3. If the column is UUID, either convert it to hex String in the query projection (SELECT lower(hex(UUIDToBytes(trace_id)))) or restore the expected schema
4. Add an ingestion-side validation that rejects spans with trace IDs that are not 16 bytes before they reach ClickHouse

Example fix

// before: UUID trace IDs in table produce dashes and fail hex.DecodeString
trace_id UUID

// after: store canonical 32-char hex string
toString(trace_id) AS trace_id  -- or change column to String and write hex.EncodeToString(id.Bytes())

var hexRe = regexp.MustCompile(`^[0-9a-fA-F]{32}$`)
func validTraceIDHex(s string) bool { return hexRe.MatchString(s) }
-- data audit query
SELECT count() FROM traces WHERE NOT match(trace_id, '^[0-9a-fA-F]{32}$');

func isValidTraceID(b []byte) bool { return len(b) == 16 }

for ids, err := range reader.FindTraceIDs(ctx, query) {
    if err != nil {
        var pathErr *hex.InvalidByteError
        if goErrors.As(err, &pathErr) || strings.Contains(err.Error(), "failed to decode trace ID") {
            log.Error("corrupt trace_id in clickhouse", "err", err)
            continue // skip result set, alert on data quality
        }
        return err
    }
}

When it happens

Trigger: Reader.FindTraceIDs returns a trace_id value that is not 32 lowercase/uppercase hex characters — e.g. an empty string, a UUID containing dashes, a base64-encoded ID, or a truncated ID produced by a custom ingestion path writing into the ClickHouse traces table.

Common situations: Another writer (custom pipeline, OTel ClickHouse exporter with a different encoding) populates the same traces table with non-hex trace IDs; schema migration converted trace_id to UUID (UUID strings contain dashes, e.g. 8-4-4-4-12); data corruption during ETL.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to append span to batch: %w
• failed to query trace IDs: %w
• failed to read trace ID rows: %w
• failed to prepare batch: %w
• failed to send batch: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/31466346d7f05800. Report an issue: GitHub.