cilium/cilium · error

Error providing Options for GRPC connection: %w

Error message

Error providing Options for GRPC connection: %w

What it means

runXDSClient in pkg/xds/experimental/client/cell.go:114 wraps the error returned by GRPCOptionsProvider.GRPCOptions(ctx) when building the gRPC dial options for the xDS connection fails. The xDS client cannot create its gRPC connection without these options (TLS credentials, etc.), so the job fails and is retried.

Source

Thrown at pkg/xds/experimental/client/cell.go:114

	in.JobGroup.Add(job.OneShot("xds-client-run", func(ctx context.Context, _ cell.Health) error {
		localNode, err := in.LocalNodeStore.Get(context.TODO())
		if err != nil {
			return fmt.Errorf("Failed to get LocalNodeStore: %w", err)
		}
		zone := localNode.Labels[core_v1.LabelTopologyZone]
		in.Log.Info("Get local node", logfields.Zone, zone)
		if zone == "" {
			return fmt.Errorf("zone is nil")
		}
		nodeID := localNode.Name
		if in.Config.NodeID != "" {
			nodeID = in.Config.NodeID
		}

		node := in.NodeBuilder.Node(nodeID, zone)
		gOps, err := in.GRPCOptionsProvider.GRPCOptions(ctx)
		if err != nil {
			return fmt.Errorf("Error providing Options for GRPC connection: %w", err)
		}
		conn, err := grpc.NewClient(in.Config.ServerAddr, gOps...)
		if err != nil {
			return fmt.Errorf("Failed to create grpc Client: %w", err)
		}
		defer conn.Close()
		in.Log.Info("Successfully run xDS client")
		return cl.Run(ctx, node, conn)
	},
		job.WithRetry(3, &job.ExponentialBackoff{Min: 1 * time.Second, Max: 5 * time.Minute}),
	))
}

View on GitHub (pinned to ac7b90affa)

Solutions

1. Inspect the wrapped %w error from the GRPC options provider to identify the failing option (usually TLS cert/key/CA loading).
2. Ensure the TLS certificates/secret for the xDS server connection are mounted and valid (not expired, correct paths).
3. The job retries 3 times with exponential backoff (1s to 5m); fix the certificate/config issue and verify the retry succeeds.
4. If using plaintext xDS, make sure the options provider is configured accordingly rather than expecting TLS material.

Example fix

// before: cert path wrong
grpcOptions:
  tlsCert: /var/run/secrets/xdscerts/tls.crt   # file missing
// after
grpcOptions:
  tlsCert: /var/lib/cilium/xds/tls.crt         # correctly mounted secret

for _, p := range []string{certPath, keyPath, caPath} {
    b, err := os.ReadFile(p)
    if err != nil {
        return fmt.Errorf("xDS TLS material %s unreadable: %w", p, err)
    }
    if p == caPath || p == certPath {
        if _, err := tls.X509KeyPair(cert, key); err != nil { /* validate parseability */ }
    }
}

err := runXDSClient(ctx)
if err != nil && strings.Contains(err.Error(), "Error providing Options for GRPC connection") {
    var te *tls.CertificateRequestError
    if errors.As(err, &te) {
        log.Error("xDS TLS certificates invalid or missing; check mounted secret", "err", err)
    }
    return err
}

When it happens

Trigger: GRPCOptions(ctx) returns an error inside the 'xds-client-run' job: the TLS secret/certificate required for the gRPC connection cannot be read or parsed, the dial-options provider is misconfigured, or its backing dependencies (e.g. secret store) are not ready or error out.

Common situations: Missing or malformed xDS TLS certificates mounted into the agent; the secret (e.g. istio ca.crt/cert chain) not yet present at startup; wrong paths configured for the credentials provider; the provider implementation returning a validation error on bad config.

Related errors

• no server TLS config is set
• cannot create hubble-relay server: %w
• xDS Server address was not provided
• Failed to create grpc Client: %w
• certificate and private key are both required, but only one

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/2d39ab0824830668. Report an issue: GitHub.