{"record":{"id":"31466346d7f05800","repo":"jaegertracing/jaeger","slug":"failed-to-decode-trace-id-w-314663","errorCode":null,"errorMessage":"failed to decode trace ID: %w","messagePattern":"failed to decode trace ID: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/v2/clickhouse/tracestore/reader.go","lineNumber":234,"sourceCode":"\t\t\terrs = append(errs, fmt.Errorf(\"failed to close rows: %w\", closeErr))\n\t\t}\n\t\tif err := errors.Join(errs...); err != nil {\n\t\t\tyield(nil, err)\n\t\t}\n\t}\n}\n\nfunc readRowIntoTraceID(rows driver.Rows) ([]tracestore.FoundTraceID, error) {\n\tvar traceIDHex string\n\tvar start, end time.Time\n\n\tif err := rows.Scan(&traceIDHex, &start, &end); err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to scan row: %w\", err)\n\t}\n\n\tb, err := hex.DecodeString(traceIDHex)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to decode trace ID: %w\", err)\n\t}\n\n\ttraceID := tracestore.FoundTraceID{\n\t\tTraceID: pcommon.TraceID(b),\n\t}\n\n\tif !start.IsZero() {\n\t\ttraceID.Start = start\n\t}\n\tif !end.IsZero() {\n\t\ttraceID.End = end\n\t}\n\n\treturn []tracestore.FoundTraceID{\n\t\ttraceID,\n\t}, nil\n}\n","sourceCodeStart":216,"sourceCodeEnd":252,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v2/clickhouse/tracestore/reader.go#L216-L252","documentation":"After scanning the trace ID as a hex string, readRowIntoTraceID decodes it with hex.DecodeString and converts the bytes to a pcommon.TraceID (16 bytes). If the string is not valid hex or its length is not 32 hex chars (16 bytes), the error is wrapped as \"failed to decode trace ID\". pcommon.TraceID requires exactly 16 bytes; shorter or longer inputs yield a garbage or invalid ID.","triggerScenarios":"Reader.FindTraceIDs returns a trace_id value that is not 32 lowercase/uppercase hex characters — e.g. an empty string, a UUID containing dashes, a base64-encoded ID, or a truncated ID produced by a custom ingestion path writing into the ClickHouse traces table.","commonSituations":"Another writer (custom pipeline, OTel ClickHouse exporter with a different encoding) populates the same traces table with non-hex trace IDs; schema migration converted trace_id to UUID (UUID strings contain dashes, e.g. 8-4-4-4-12); data corruption during ETL.","solutions":["Inspect the offending rows: SELECT trace_id FROM traces WHERE NOT match(trace_id, '^[0-9a-fA-F]{32}$') to find malformed IDs","Ensure all writers into the table encode trace IDs as 32-char lowercase hex (UTF8BytesString of the 16-byte ID)","If the column is UUID, either convert it to hex String in the query projection (SELECT lower(hex(UUIDToBytes(trace_id)))) or restore the expected schema","Add an ingestion-side validation that rejects spans with trace IDs that are not 16 bytes before they reach ClickHouse"],"exampleFix":"// before: UUID trace IDs in table produce dashes and fail hex.DecodeString\ntrace_id UUID\n\n// after: store canonical 32-char hex string\ntoString(trace_id) AS trace_id  -- or change column to String and write hex.EncodeToString(id.Bytes())","handlingStrategy":"validation","validationCode":"var hexRe = regexp.MustCompile(`^[0-9a-fA-F]{32}$`)\nfunc validTraceIDHex(s string) bool { return hexRe.MatchString(s) }\n-- data audit query\nSELECT count() FROM traces WHERE NOT match(trace_id, '^[0-9a-fA-F]{32}$');","typeGuard":"func isValidTraceID(b []byte) bool { return len(b) == 16 }","tryCatchPattern":"for ids, err := range reader.FindTraceIDs(ctx, query) {\n    if err != nil {\n        var pathErr *hex.InvalidByteError\n        if goErrors.As(err, &pathErr) || strings.Contains(err.Error(), \"failed to decode trace ID\") {\n            log.Error(\"corrupt trace_id in clickhouse\", \"err\", err)\n            continue // skip result set, alert on data quality\n        }\n        return err\n    }\n}","preventionTips":["Ensure all writers encode trace IDs as 32-char lowercase hex, never UUID strings","Reject spans with non-16-byte trace IDs at ingestion time","Audit the traces table periodically with the regex match query for malformed IDs","Prevent foreign exporters from writing into the Jaeger-owned traces table"],"tags":["clickhouse","go","data-integrity","hex-decode"],"backgroundTag":"invalid-hex-trace-id","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}