jaegertracing/jaeger · critical

failed to query services: %w

Error message

failed to query services: %w

What it means

Reader.GetServices returns this when the conn.Query call executing sql.SelectServices fails before any rows are read. It wraps the clickhouse-go driver error, so the root cause is at the connection, authentication, or server-rejection layer, not in result decoding.

Source

Thrown at internal/storage/v2/clickhouse/tracestore/reader.go:113

			}
			if rowsErr := rows.Err(); rowsErr != nil {
				errs = append(errs, fmt.Errorf("failed to read span rows: %w", rowsErr))
			}
			if closeErr := rows.Close(); closeErr != nil {
				errs = append(errs, fmt.Errorf("failed to close rows: %w", closeErr))
			}
			if err := errors.Join(errs...); err != nil {
				yield(nil, err)
				return
			}
		}
	}
}

func (r *Reader) GetServices(ctx context.Context) ([]string, error) {
	rows, err := r.conn.Query(ctx, sql.SelectServices)
	if err != nil {
		return nil, fmt.Errorf("failed to query services: %w", err)
	}

	var (
		services []string
		errs     []error
	)
	for rows.Next() {
		var service dbmodel.Service
		if scanErr := rows.ScanStruct(&service); scanErr != nil {
			errs = append(errs, fmt.Errorf("failed to scan row: %w", scanErr))
			break
		}
		services = append(services, service.Name)
	}
	if rowsErr := rows.Err(); rowsErr != nil {
		errs = append(errs, fmt.Errorf("failed to read service rows: %w", rowsErr))
	}
	if closeErr := rows.Close(); closeErr != nil {

View on GitHub (pinned to 806f444784)

Solutions

1. Run the sql.SelectServices query manually with the same credentials to see the server's error
2. Verify DSN configuration (host, port, database, auth) in the jaeger storage config
3. Apply schema migrations so the services table exists
4. GRANT SELECT on the required tables to the jaeger ClickHouse user

Example fix

// before
GRANT SELECT ON default.traces TO jaeger_user; -- services table not granted
// after
GRANT SELECT ON default.* TO jaeger_user; -- or grant the specific services table

// preflight services query availability
if err := conn.Ping(ctx); err != nil {
	return fmt.Errorf("clickhouse unreachable: %w", err)
}
// and confirm the target table exists
var exists uint8
conn.QueryRow(ctx, "EXISTS TABLE service_names").Scan(&exists)

services, err := reader.GetServices(ctx)
if err != nil {
	if strings.Contains(err.Error(), "failed to query services") {
		return cachedServices, nil // serve stale list while ClickHouse recovers
	}
	return nil, err
}

When it happens

Trigger: Calling GetServices while ClickHouse is down/unreachable, credentials are invalid, the service_names table (or whatever sql.SelectServices targets) is missing, or the user lacks SELECT permission.

Common situations: Fresh deployment where schema/migrations weren't applied; wrong DSN in jaeger storage config; ClickHouse user lacking grants on the target table; DNS/network issues between jaeger and the ClickHouse cluster.

Related errors

• failed to query trace: %w
• error reading service_names from storage: %w
• failed to query dependencies: %w
• failed to query trace IDs: %w
• failed to prepare batch: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/965f48466f42a546. Report an issue: GitHub.