jaegertracing/jaeger · critical

failed to query trace IDs: %w

Error message

failed to query trace IDs: %w

What it means

After building the query, FindTraceIDs executes it with r.conn.Query. If the clickhouse-go v2 driver cannot execute the query — connection refused/dropped, authentication failure, DNS resolution, server-side SQL error, or context cancellation — the iterator yields the error wrapped as "failed to query trace IDs". This is the network/server-bound failure point of the trace-ID search path.

Source

Thrown at internal/storage/v2/clickhouse/tracestore/reader.go:266

	return []tracestore.FoundTraceID{
		traceID,
	}, nil
}

func (r *Reader) FindTraceIDs(
	ctx context.Context,
	query tracestore.TraceQueryParams,
) iter.Seq2[[]tracestore.FoundTraceID, error] {
	return func(yield func([]tracestore.FoundTraceID, error) bool) {
		q, args, err := r.buildFindTraceIDsQuery(ctx, query)
		if err != nil {
			yield(nil, fmt.Errorf("failed to build query: %w", err))
			return
		}

		rows, err := r.conn.Query(ctx, q, args...)
		if err != nil {
			yield(nil, fmt.Errorf("failed to query trace IDs: %w", err))
			return
		}

		var errs []error
		for rows.Next() {
			traceID, scanErr := readRowIntoTraceID(rows)
			if scanErr != nil {
				errs = append(errs, scanErr)
				break
			}
			if !yield(traceID, nil) {
				_ = rows.Close()
				return
			}
		}
		if rowsErr := rows.Err(); rowsErr != nil {
			errs = append(errs, fmt.Errorf("failed to read trace ID rows: %w", rowsErr))
		}

View on GitHub (pinned to 806f444784)

Solutions

1. Check connectivity from the Jaeger host: clickhouse-client --host <host> --port <port> --user <user> --password with the same credentials as the DSN
2. Verify the DSN (clickhouse://host:port/db?username=...&password=...&secure=...) in the storage config, including database name and TLS settings
3. Confirm the traces table exists: SHOW TABLES FROM <db>; run the Jaeger ClickHouse schema migrations if not
4. Look at the wrapped error via errors.Unwrap — clickhouse-go returns typed errors (ErrServerClosed, auth errors) that identify auth vs network vs SQL
5. Check ClickHouse server logs for the rejected query and increase client-side timeout if the query is being killed by context deadline

Example fix

// before: wrong port / no auth in DSN
dsn := "clickhouse://localhost:9009"

// after: correct port, credentials, and database
dsn := "clickhouse://localhost:9000/traces?username=default&password=secret&dial_timeout=10s"

u, err := url.Parse(dsn)
if err != nil { return err }
if err := pingClickHouse(ctx, dsn); err != nil { return fmt.Errorf("clickhouse unreachable: %w", err) } // SELECT 1 with same creds

for ids, err := range reader.FindTraceIDs(ctx, query) {
    if err != nil {
        if strings.Contains(err.Error(), "failed to query trace IDs") {
            if isTransient(err) { // net.Error timeout, connection reset
                return retryWithBackoff(ctx, query)
            }
            return err // auth/SQL errors are not retryable
        }
        return err
    }
}

When it happens

Trigger: Calling Reader.FindTraceIDs when: the ClickHouse endpoint is unreachable (wrong DSN host/port), credentials in the DSN are wrong (authentication failed in native/HTTP protocol), the database does not exist, the server rejects the SQL (unknown table/column), or ctx is canceled/timed out before execution.

Common situations: ClickHouse not started or wrong port in jaeger clickhouse DSN config; password changed and config not updated; database name mismatch after redeploy; schema migrations not run so the traces table is missing; LB/proxy in front of ClickHouse dropped the connection; TLS config mismatch.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• failed to send batch: %w
• failed to query trace: %w
• failed to read trace ID rows: %w
• failed to prepare batch: %w
• websocket dial %s: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/a1fa83b2b3eac618. Report an issue: GitHub.