jaegertracing/jaeger · error

failed to read span rows: %w

Error message

failed to read span rows: %w

What it means

After scanning rows, Reader.GetTraces checks rows.Err() from the clickhouse driver; a non-nil result means the row iteration itself failed (e.g. the connection dropped mid-result-set, the server aborted the query, or a network timeout occurred while streaming). The error is collected and joined with any close error before being yielded.

Source

Thrown at internal/storage/v2/clickhouse/tracestore/reader.go:97

				yield(nil, fmt.Errorf("failed to query trace: %w", err))
				return
			}

			var errs []error
			for rows.Next() {
				span, scanErr := dbmodel.ScanRow(rows)
				if scanErr != nil {
					errs = append(errs, fmt.Errorf("failed to scan span row: %w", scanErr))
					break
				}
				trace := dbmodel.FromRow(span)
				if !yield([]ptrace.Traces{trace}, nil) {
					_ = rows.Close()
					return
				}
			}
			if rowsErr := rows.Err(); rowsErr != nil {
				errs = append(errs, fmt.Errorf("failed to read span rows: %w", rowsErr))
			}
			if closeErr := rows.Close(); closeErr != nil {
				errs = append(errs, fmt.Errorf("failed to close rows: %w", closeErr))
			}
			if err := errors.Join(errs...); err != nil {
				yield(nil, err)
				return
			}
		}
	}
}

func (r *Reader) GetServices(ctx context.Context) ([]string, error) {
	rows, err := r.conn.Query(ctx, sql.SelectServices)
	if err != nil {
		return nil, fmt.Errorf("failed to query services: %w", err)
	}

View on GitHub (pinned to 806f444784)

Solutions

1. Check the wrapped driver error for timeout vs connection-reset and adjust ClickHouse/LB timeouts accordingly
2. Reduce query scope (narrow time range, smaller traces) or increase max_execution_time
3. Enable connection keepalive/health checks in the clickhouse-go dial settings
4. Retry GetTraces; the iterator pattern makes re-issuing the per-trace query safe

Example fix

// before (default dial settings, LB kills idle conn)
conn, err := clickhouse.Open(&clickhouse.Options{Addr: []string{addr}})
// after
conn, err := clickhouse.Open(&clickhouse.Options{
	Addr: []string{addr},
	DialTimeout: 30 * time.Second,
	Settings: clickhouse.Settings{"max_execution_time": 120},
})

// preflight with a small bounded query to ensure streaming works
rows, err := conn.Query(ctx, "SELECT 1")
if err == nil { rows.Close() }

for traces, err := range reader.GetTraces(ctx, ids) {
	if err != nil && strings.Contains(err.Error(), "failed to read span rows") {
		// connection dropped mid-stream — retry the trace once with backoff
		continue
	}
}

When it happens

Trigger: Calling GetTraces on a large trace where the underlying ClickHouse connection is reset, times out, or the server cancels the query while rows are still being streamed.

Common situations: Long-running queries hitting idle-connection/network timeouts (LB idle cutoffs, NAT drops); ClickHouse max_execution_time exceeded; server restart during a large scan.

Related errors

• failed to read trace ID rows: %w
• failed to get attribute metadata: %w
• failed to query trace: %w
• failed to query trace IDs: %w
• failed to send batch: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/d10174ecf1b41f0f. Report an issue: GitHub.