{"record":{"id":"a1fa83b2b3eac618","repo":"jaegertracing/jaeger","slug":"failed-to-query-trace-ids-w","errorCode":null,"errorMessage":"failed to query trace IDs: %w","messagePattern":"failed to query trace IDs: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"internal/storage/v2/clickhouse/tracestore/reader.go","lineNumber":266,"sourceCode":"\treturn []tracestore.FoundTraceID{\n\t\ttraceID,\n\t}, nil\n}\n\nfunc (r *Reader) FindTraceIDs(\n\tctx context.Context,\n\tquery tracestore.TraceQueryParams,\n) iter.Seq2[[]tracestore.FoundTraceID, error] {\n\treturn func(yield func([]tracestore.FoundTraceID, error) bool) {\n\t\tq, args, err := r.buildFindTraceIDsQuery(ctx, query)\n\t\tif err != nil {\n\t\t\tyield(nil, fmt.Errorf(\"failed to build query: %w\", err))\n\t\t\treturn\n\t\t}\n\n\t\trows, err := r.conn.Query(ctx, q, args...)\n\t\tif err != nil {\n\t\t\tyield(nil, fmt.Errorf(\"failed to query trace IDs: %w\", err))\n\t\t\treturn\n\t\t}\n\n\t\tvar errs []error\n\t\tfor rows.Next() {\n\t\t\ttraceID, scanErr := readRowIntoTraceID(rows)\n\t\t\tif scanErr != nil {\n\t\t\t\terrs = append(errs, scanErr)\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tif !yield(traceID, nil) {\n\t\t\t\t_ = rows.Close()\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t\tif rowsErr := rows.Err(); rowsErr != nil {\n\t\t\terrs = append(errs, fmt.Errorf(\"failed to read trace ID rows: %w\", rowsErr))\n\t\t}","sourceCodeStart":248,"sourceCodeEnd":284,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/v2/clickhouse/tracestore/reader.go#L248-L284","documentation":"After building the query, FindTraceIDs executes it with r.conn.Query. If the clickhouse-go v2 driver cannot execute the query — connection refused/dropped, authentication failure, DNS resolution, server-side SQL error, or context cancellation — the iterator yields the error wrapped as \"failed to query trace IDs\". This is the network/server-bound failure point of the trace-ID search path.","triggerScenarios":"Calling Reader.FindTraceIDs when: the ClickHouse endpoint is unreachable (wrong DSN host/port), credentials in the DSN are wrong (authentication failed in native/HTTP protocol), the database does not exist, the server rejects the SQL (unknown table/column), or ctx is canceled/timed out before execution.","commonSituations":"ClickHouse not started or wrong port in jaeger clickhouse DSN config; password changed and config not updated; database name mismatch after redeploy; schema migrations not run so the traces table is missing; LB/proxy in front of ClickHouse dropped the connection; TLS config mismatch.","solutions":["Check connectivity from the Jaeger host: clickhouse-client --host <host> --port <port> --user <user> --password with the same credentials as the DSN","Verify the DSN (clickhouse://host:port/db?username=...&password=...&secure=...) in the storage config, including database name and TLS settings","Confirm the traces table exists: SHOW TABLES FROM <db>; run the Jaeger ClickHouse schema migrations if not","Look at the wrapped error via errors.Unwrap — clickhouse-go returns typed errors (ErrServerClosed, auth errors) that identify auth vs network vs SQL","Check ClickHouse server logs for the rejected query and increase client-side timeout if the query is being killed by context deadline"],"exampleFix":"// before: wrong port / no auth in DSN\ndsn := \"clickhouse://localhost:9009\"\n\n// after: correct port, credentials, and database\ndsn := \"clickhouse://localhost:9000/traces?username=default&password=secret&dial_timeout=10s\"","handlingStrategy":"retry","validationCode":"u, err := url.Parse(dsn)\nif err != nil { return err }\nif err := pingClickHouse(ctx, dsn); err != nil { return fmt.Errorf(\"clickhouse unreachable: %w\", err) } // SELECT 1 with same creds","typeGuard":null,"tryCatchPattern":"for ids, err := range reader.FindTraceIDs(ctx, query) {\n    if err != nil {\n        if strings.Contains(err.Error(), \"failed to query trace IDs\") {\n            if isTransient(err) { // net.Error timeout, connection reset\n                return retryWithBackoff(ctx, query)\n            }\n            return err // auth/SQL errors are not retryable\n        }\n        return err\n    }\n}","preventionTips":["Add a startup health check (SELECT 1) with the production DSN","Set dial_timeout/read_timeout in the DSN to realistic values","Keep schema migrations in the deployment pipeline so tables always exist","Monitor ClickHouse availability and alert before users search traces","Rotate credentials in config and secret store simultaneously"],"tags":["clickhouse","go","network","database","connection"],"backgroundTag":"connection-refused","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}