{"record":{"id":"d10174ecf1b41f0f","repo":"jaegertracing/jaeger","slug":"failed-to-read-span-rows-w","errorCode":null,"errorMessage":"failed to read span rows: %w","messagePattern":"failed to read span rows: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/v2/clickhouse/tracestore/reader.go","lineNumber":97,"sourceCode":"\t\t\t\tyield(nil, fmt.Errorf(\"failed to query trace: %w\", err))\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar errs []error\n\t\t\tfor rows.Next() {\n\t\t\t\tspan, scanErr := dbmodel.ScanRow(rows)\n\t\t\t\tif scanErr != nil {\n\t\t\t\t\terrs = append(errs, fmt.Errorf(\"failed to scan span row: %w\", scanErr))\n\t\t\t\t\tbreak\n\t\t\t\t}\n\t\t\t\ttrace := dbmodel.FromRow(span)\n\t\t\t\tif !yield([]ptrace.Traces{trace}, nil) {\n\t\t\t\t\t_ = rows.Close()\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t}\n\t\t\tif rowsErr := rows.Err(); rowsErr != nil {\n\t\t\t\terrs = append(errs, fmt.Errorf(\"failed to read span rows: %w\", rowsErr))\n\t\t\t}\n\t\t\tif closeErr := rows.Close(); closeErr != nil {\n\t\t\t\terrs = append(errs, fmt.Errorf(\"failed to close rows: %w\", closeErr))\n\t\t\t}\n\t\t\tif err := errors.Join(errs...); err != nil {\n\t\t\t\tyield(nil, err)\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc (r *Reader) GetServices(ctx context.Context) ([]string, error) {\n\trows, err := r.conn.Query(ctx, sql.SelectServices)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to query services: %w\", err)\n\t}\n","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v2/clickhouse/tracestore/reader.go#L79-L115","documentation":"After scanning rows, Reader.GetTraces checks rows.Err() from the clickhouse driver; a non-nil result means the row iteration itself failed (e.g. the connection dropped mid-result-set, the server aborted the query, or a network timeout occurred while streaming). The error is collected and joined with any close error before being yielded.","triggerScenarios":"Calling GetTraces on a large trace where the underlying ClickHouse connection is reset, times out, or the server cancels the query while rows are still being streamed.","commonSituations":"Long-running queries hitting idle-connection/network timeouts (LB idle cutoffs, NAT drops); ClickHouse max_execution_time exceeded; server restart during a large scan.","solutions":["Check the wrapped driver error for timeout vs connection-reset and adjust ClickHouse/LB timeouts accordingly","Reduce query scope (narrow time range, smaller traces) or increase max_execution_time","Enable connection keepalive/health checks in the clickhouse-go dial settings","Retry GetTraces; the iterator pattern makes re-issuing the per-trace query safe"],"exampleFix":"// before (default dial settings, LB kills idle conn)\nconn, err := clickhouse.Open(&clickhouse.Options{Addr: []string{addr}})\n// after\nconn, err := clickhouse.Open(&clickhouse.Options{\n\tAddr: []string{addr},\n\tDialTimeout: 30 * time.Second,\n\tSettings: clickhouse.Settings{\"max_execution_time\": 120},\n})","handlingStrategy":"retry","validationCode":"// preflight with a small bounded query to ensure streaming works\nrows, err := conn.Query(ctx, \"SELECT 1\")\nif err == nil { rows.Close() }","typeGuard":null,"tryCatchPattern":"for traces, err := range reader.GetTraces(ctx, ids) {\n\tif err != nil && strings.Contains(err.Error(), \"failed to read span rows\") {\n\t\t// connection dropped mid-stream — retry the trace once with backoff\n\t\tcontinue\n\t}\n}","preventionTips":["Set max_execution_time and client timeouts larger than your largest expected trace scan","Configure LBs/NATs with idle timeouts longer than long query durations","Enable clickhouse-go compression and keepalives for long result streams","Split very large trace retrievals across time-bounded queries"],"tags":["clickhouse","network","timeout","streaming"],"backgroundTag":"connection-reset-during-query","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}