jaegertracing/jaeger · error

failed to read trace ID rows: %w

Error message

failed to read trace ID rows: %w

What it means

After iterating all trace-ID rows, FindTraceIDs checks rows.Err() from the clickhouse-go driver; a non-nil value means the row stream terminated abnormally (connection loss, server abort, protocol error mid-stream) rather than reaching a clean EOF. The error is wrapped as "failed to read trace ID rows" and joined with any other iteration errors before being yielded. It indicates partial results: some trace IDs may have been yielded before the failure.

Source

Thrown at internal/storage/v2/clickhouse/tracestore/reader.go:283

		if err != nil {
			yield(nil, fmt.Errorf("failed to query trace IDs: %w", err))
			return
		}

		var errs []error
		for rows.Next() {
			traceID, scanErr := readRowIntoTraceID(rows)
			if scanErr != nil {
				errs = append(errs, scanErr)
				break
			}
			if !yield(traceID, nil) {
				_ = rows.Close()
				return
			}
		}
		if rowsErr := rows.Err(); rowsErr != nil {
			errs = append(errs, fmt.Errorf("failed to read trace ID rows: %w", rowsErr))
		}
		if closeErr := rows.Close(); closeErr != nil {
			errs = append(errs, fmt.Errorf("failed to close rows: %w", closeErr))
		}
		if err := errors.Join(errs...); err != nil {
			yield(nil, err)
		}
	}
}

View on GitHub (pinned to 806f444784)

Solutions

1. Check ClickHouse server logs for query abort reasons (max_execution_time, memory limits) and raise the corresponding server limits
2. Narrow the search time range / add limits in TraceQueryParams so streams finish quickly
3. Increase the context deadline or driver settings (dial_timeout, read_timeout) in the DSN
4. Tune proxy/LB keepalive and idle timeouts to exceed the longest expected query duration
5. Retry the query — the iterator contract allows retrying; treat already-yielded partial IDs as discarded

Example fix

// before: unbounded scan aborted by max_execution_time
query := tracestore.TraceQueryParams{StartTimeMin: time.Time{}, StartTimeMax: time.Now()} // years of data

// after: chunk the time range and set server-friendly limits
query := tracestore.TraceQueryParams{StartTimeMin: start, StartTimeMax: start.Add(24 * time.Hour)}

if dl, ok := ctx.Deadline(); ok && time.Until(dl) < 30*time.Second {
    return errors.New("deadline too short for trace-id scan, extend context deadline")
}

err := retry.Do(func() error {
    for ids, e := range reader.FindTraceIDs(ctx, query) {
        if e != nil {
            if strings.Contains(e.Error(), "failed to read trace ID rows") {
                return e // triggers retry with fresh connection
            }
            return retry.Unrecoverable(e)
        }
        collect(ids)
    }
    return nil
}, retry.Attempts(3), retry.Delay(time.Second))

When it happens

Trigger: Calling Reader.FindTraceIDs over a large result set when the ClickHouse connection dies mid-iteration — server restart, network partition, proxy idle timeout, context cancellation between rows, or ClickHouse killing the query (max_execution_time exceeded).

Common situations: Long-running scan over a wide time range hitting ClickHouse's max_execution_time; LB (ChProxy/HAProxy) dropping idle keepalive connections during slow row consumption; OOM kill of the ClickHouse server under heavy concurrent queries.

Related errors

• failed to read span rows: %w
• failed to query trace IDs: %w
• failed to send batch: %w
• could not start jaeger-query: %w
• failed to get attribute metadata: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/39d0612f2ceb131b. Report an issue: GitHub.