vxcontrol/pentagi · error

failed to create telemetry connection: %w

Error message

failed to create telemetry connection: %w

What it means

NewTelemetryClient failed at the grpc.NewClient step that creates the shared non-blocking gRPC connection to the OTLP collector. Because grpc.NewClient is lazy, failure here almost always means an invalid target string (bad address/URL syntax), not an unreachable server.

Source

Thrown at backend/pkg/observability/otelclient.go:113

	return errors.Join(errs...)
}

func NewTelemetryClient(ctx context.Context, cfg *config.Config) (TelemetryClient, error) {
	if cfg.TelemetryEndpoint == "" {
		return nil, fmt.Errorf("telemetry endpoint is not set: %w", ErrNotConfigured)
	}

	// grpc.NewClient is non-blocking: it never dials during startup, so a
	// set-but-unreachable collector can't stall main(), and the connection is
	// established (and re-established) lazily in the background — a collector that
	// comes up after the app does connects on its own, without a restart.
	conn, err := grpc.NewClient(
		cfg.TelemetryEndpoint,
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithDefaultCallOptions(grpc.WaitForReady(true)),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create telemetry connection: %w", err)
	}

	// Build all three exporters before creating any provider. Exporter creation
	// is the only remaining error path, and at that point no batch/reader
	// goroutine has started yet, so closing the connection is a complete teardown.
	logExporter, err := otlploggrpc.New(ctx, otlploggrpc.WithGRPCConn(conn))
	if err != nil {
		_ = conn.Close()
		return nil, fmt.Errorf("failed to create log exporter: %w", err)
	}
	metricExporter, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithGRPCConn(conn))
	if err != nil {
		_ = conn.Close()
		return nil, fmt.Errorf("failed to create metric exporter: %w", err)
	}
	spanExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
	if err != nil {
		_ = conn.Close()

View on GitHub (pinned to ea665308ba)

Solutions

1. Ensure cfg.TelemetryEndpoint is plain host:port (e.g. 'otel-collector:4317'), no scheme, no path
2. Check the wrapped error message for target-parse details
3. Fix the endpoint env var in .env / docker-compose environment and restart
4. If using DNS, make sure the resolver works inside the container network

Example fix

// before
TELEMETRY_ENDPOINT=https://localhost:4317/v1/metrics
// after
TELEMETRY_ENDPOINT=localhost:4317

endpoint := cfg.TelemetryEndpoint
if endpoint == "" || strings.Contains(endpoint, "://") || strings.Contains(endpoint, "/") {
    return fmt.Errorf("invalid telemetry endpoint %q: want host:port", endpoint)
}

client, err := NewTelemetryClient(ctx, cfg)
if err != nil {
    if strings.Contains(err.Error(), "failed to create telemetry connection") {
        return fmt.Errorf("check TELEMETRY_ENDPOINT format (host:port, no scheme): %w", err)
    }
    return err
}

When it happens

Trigger: cfg.TelemetryEndpoint contains a malformed target (e.g. includes a scheme like http://, spaces, or an unparseable host:port), or gRPC client construction fails for a credentials/options reason.

Common situations: Users pasting a full URL (https://collector:4317) instead of host:port; trailing slashes or protocol prefixes in the telemetry endpoint env var.

Related errors

• failed to force flush logger: %w
• failed to force flush meter: %w
• failed to force flush tracer: %w
• telemetry endpoint is not set: %w
• failed to create log exporter: %w

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/350ef83a6d41f468. Report an issue: GitHub.