{"record":{"id":"11e671eb7cd0b1c6","repo":"jaegertracing/jaeger","slug":"failed-to-query-trace-w","errorCode":null,"errorMessage":"failed to query trace: %w","messagePattern":"failed to query trace: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"internal/storage/v2/clickhouse/tracestore/reader.go","lineNumber":79,"sourceCode":"func (*Reader) SearchCapabilities(context.Context) (tracestore.SearchCapabilities, error) {\n\treturn tracestore.SearchCapabilities{\n\t\t// The search SQL starts from \"WHERE 1=1\" and appends the service predicate only\n\t\t// when the query carries a name, so an omitted name matches spans from every\n\t\t// service.\n\t\tWithoutServiceName: true,\n\t}, nil\n}\n\nfunc (r *Reader) GetTraces(\n\tctx context.Context,\n\ttraceIDs ...tracestore.GetTraceParams,\n) iter.Seq2[[]ptrace.Traces, error] {\n\treturn func(yield func([]ptrace.Traces, error) bool) {\n\t\tfor _, traceID := range traceIDs {\n\t\t\tquery, args := buildGetTracesQuery(traceID)\n\t\t\trows, err := r.conn.Query(ctx, query, args...)\n\t\t\tif err != nil {\n\t\t\t\tyield(nil, fmt.Errorf(\"failed to query trace: %w\", err))\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar errs []error\n\t\t\tfor rows.Next() {\n\t\t\t\tspan, scanErr := dbmodel.ScanRow(rows)\n\t\t\t\tif scanErr != nil {\n\t\t\t\t\terrs = append(errs, fmt.Errorf(\"failed to scan span row: %w\", scanErr))\n\t\t\t\t\tbreak\n\t\t\t\t}\n\t\t\t\ttrace := dbmodel.FromRow(span)\n\t\t\t\tif !yield([]ptrace.Traces{trace}, nil) {\n\t\t\t\t\t_ = rows.Close()\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t}\n\t\t\tif rowsErr := rows.Err(); rowsErr != nil {\n\t\t\t\terrs = append(errs, fmt.Errorf(\"failed to read span rows: %w\", rowsErr))","sourceCodeStart":61,"sourceCodeEnd":97,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v2/clickhouse/tracestore/reader.go#L61-L97","documentation":"Reader.GetTraces yields this error when the clickhouse-go conn.Query call for a single trace ID fails. This happens before any rows are read, so it reflects query construction, connection, or server-side rejection of the SELECT built by buildGetTracesQuery. The iterator stops immediately and surfaces the underlying driver error wrapped with %w.","triggerScenarios":"Calling GetTraces with a trace ID while ClickHouse is unreachable, the connection pool is exhausted, credentials are wrong, the database/table is missing, or the generated SQL is rejected by the server (syntax/permissions).","commonSituations":"ClickHouse pod restarted or network partition in k8s; misconfigured DSN (wrong host/port/database) in the storage config; schema migrations not applied so the spans table doesn't exist; TLS/auth mismatch between client and server.","solutions":["Verify ClickHouse connectivity (host, port, credentials, TLS) using the configured DSN, e.g. with clickhouse-client","Confirm the spans table exists and migrations were applied","Inspect the wrapped driver error (errors.Unwrap / %v of the yielded error) for the precise server message","Add retry/backoff around GetTraces for transient network failures"],"exampleFix":"// before (no retry, hard failure on transient error)\ntraces, err := reader.GetTraces(ctx, ids)\n// after\ntraces, err := retry.Do(ctx, func() (iter.Seq2[[]ptrace.Traces, error], error) {\n\treturn reader.GetTraces(ctx, ids)\n}, retry.Attempts(3))","handlingStrategy":"retry","validationCode":"// preflight connectivity before querying\nconn := clickhouse.Open(&clickhouse.Options{Addr: []string{addr}})\nif err := conn.Ping(ctx); err != nil {\n\treturn fmt.Errorf(\"clickhouse unreachable: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"for traces, err := range reader.GetTraces(ctx, ids) {\n\tif err != nil {\n\t\tif strings.Contains(err.Error(), \"failed to query trace\") {\n\t\t\t// transient driver failure — retry with backoff\n\t\t\tcontinue\n\t\t}\n\t\treturn err\n\t}\n\tprocess(traces)\n}","preventionTips":["Ping the ClickHouse connection at startup and periodically","Use a DSN with explicit timeouts and connection-pool settings","Ensure schema migrations run before jaeger starts","Monitor ClickHouse availability with alerts on query failure rates"],"tags":["clickhouse","database","query","network"],"backgroundTag":"clickhouse-query-failed","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}