{"record":{"id":"39d0612f2ceb131b","repo":"jaegertracing/jaeger","slug":"failed-to-read-trace-id-rows-w","errorCode":null,"errorMessage":"failed to read trace ID rows: %w","messagePattern":"failed to read trace ID rows: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/v2/clickhouse/tracestore/reader.go","lineNumber":283,"sourceCode":"\t\tif err != nil {\n\t\t\tyield(nil, fmt.Errorf(\"failed to query trace IDs: %w\", err))\n\t\t\treturn\n\t\t}\n\n\t\tvar errs []error\n\t\tfor rows.Next() {\n\t\t\ttraceID, scanErr := readRowIntoTraceID(rows)\n\t\t\tif scanErr != nil {\n\t\t\t\terrs = append(errs, scanErr)\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tif !yield(traceID, nil) {\n\t\t\t\t_ = rows.Close()\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t\tif rowsErr := rows.Err(); rowsErr != nil {\n\t\t\terrs = append(errs, fmt.Errorf(\"failed to read trace ID rows: %w\", rowsErr))\n\t\t}\n\t\tif closeErr := rows.Close(); closeErr != nil {\n\t\t\terrs = append(errs, fmt.Errorf(\"failed to close rows: %w\", closeErr))\n\t\t}\n\t\tif err := errors.Join(errs...); err != nil {\n\t\t\tyield(nil, err)\n\t\t}\n\t}\n}\n","sourceCodeStart":265,"sourceCodeEnd":293,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v2/clickhouse/tracestore/reader.go#L265-L293","documentation":"After iterating all trace-ID rows, FindTraceIDs checks rows.Err() from the clickhouse-go driver; a non-nil value means the row stream terminated abnormally (connection loss, server abort, protocol error mid-stream) rather than reaching a clean EOF. The error is wrapped as \"failed to read trace ID rows\" and joined with any other iteration errors before being yielded. It indicates partial results: some trace IDs may have been yielded before the failure.","triggerScenarios":"Calling Reader.FindTraceIDs over a large result set when the ClickHouse connection dies mid-iteration — server restart, network partition, proxy idle timeout, context cancellation between rows, or ClickHouse killing the query (max_execution_time exceeded).","commonSituations":"Long-running scan over a wide time range hitting ClickHouse's max_execution_time; LB (ChProxy/HAProxy) dropping idle keepalive connections during slow row consumption; OOM kill of the ClickHouse server under heavy concurrent queries.","solutions":["Check ClickHouse server logs for query abort reasons (max_execution_time, memory limits) and raise the corresponding server limits","Narrow the search time range / add limits in TraceQueryParams so streams finish quickly","Increase the context deadline or driver settings (dial_timeout, read_timeout) in the DSN","Tune proxy/LB keepalive and idle timeouts to exceed the longest expected query duration","Retry the query — the iterator contract allows retrying; treat already-yielded partial IDs as discarded"],"exampleFix":"// before: unbounded scan aborted by max_execution_time\nquery := tracestore.TraceQueryParams{StartTimeMin: time.Time{}, StartTimeMax: time.Now()} // years of data\n\n// after: chunk the time range and set server-friendly limits\nquery := tracestore.TraceQueryParams{StartTimeMin: start, StartTimeMax: start.Add(24 * time.Hour)}","handlingStrategy":"retry","validationCode":"if dl, ok := ctx.Deadline(); ok && time.Until(dl) < 30*time.Second {\n    return errors.New(\"deadline too short for trace-id scan, extend context deadline\")\n}","typeGuard":null,"tryCatchPattern":"err := retry.Do(func() error {\n    for ids, e := range reader.FindTraceIDs(ctx, query) {\n        if e != nil {\n            if strings.Contains(e.Error(), \"failed to read trace ID rows\") {\n                return e // triggers retry with fresh connection\n            }\n            return retry.Unrecoverable(e)\n        }\n        collect(ids)\n    }\n    return nil\n}, retry.Attempts(3), retry.Delay(time.Second))","preventionTips":["Chunk large time ranges into smaller scans so streams finish quickly","Raise ClickHouse max_execution_time and memory limits for search workloads","Set LB/proxy idle timeouts above the longest expected query duration","Discard partially yielded IDs when retrying — treat each attempt as atomic"],"tags":["clickhouse","go","network","streaming"],"backgroundTag":"connection-reset-during-query","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}