xds: fetching trusted roots from CertificateProvider failed

Error message

xds: fetching trusted roots from CertificateProvider failed: %v

What it means

Raised on the client side during xDS-driven TLS when hi.rootProvider.KeyMaterial(ctx) returns an error. The root certificate provider (typically a file-watcher or xDS-cert-provider plugin) could not supply the trusted root CA material needed to verify the server.

Solutions

Check the CertificateProvider config (file path / xDS resource name) and that the root bundle exists and is readable.
Inspect the wrapped error for the provider-specific cause (file not found, parse error, context cancelled).
Verify the xDS management server is sending a CertificateProviderInstance with root certs for this cluster.
Confirm the provider is not closed before handshake and the context is not already cancelled.
Ensure the CA bundle file is valid PEM and not empty.

Example fix

// before: root cert file path missing -> provider.KeyMaterial errors
// after: mount /etc/grpc/certs/ca.pem and configure file-watcher provider to that path

Defensive patterns

Strategy: try-catch

Validate before calling

func rootProviderHealthy(p certprovider.Provider) bool {
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
    defer cancel()
    km, err := p.KeyMaterial(ctx)
    return err == nil && km != nil && km.Roots != nil && len(km.Roots.Subjects()) > 0
}

Try / catch

In ClientSideTLSConfig's caller, distinguish useFallback from err; if err mentions 'fetching trusted roots', surface it as a config/secret problem and fail fast rather than retrying in a hot loop.

Prevention

Mount root CA bundles via a provisioned secret and validate presence at startup.
Health-check the certificate provider before serving traffic.
Alert on provider KeyMaterial errors so rotation gaps are caught early.

When it happens

Trigger: The root cert file referenced by the CertificateProvider is missing, unreadable, or malformed; the xDS control plane did not deliver a CertificateProviderInstance for roots; the provider was closed or its context was cancelled; refresh from the management server failed.

Common situations: Secret mount path wrong or empty in the pod; mTLS root bundle not yet rotated in; xDS LDS/CDS resource lacks the security config; provider plugin version mismatch; filesystem permission error reading the CA bundle.

Understand the failure class

SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

AI-assisted analysis of grpc/grpc-go@0c51461d27 (2026-08-11). Data as JSON: /api/errors/78555b530c6f9186. Report an issue: GitHub.

Appendix: source

Thrown at internal/credentials/xds/handshake_info.go:224

	// On the client side, rootProvider is mandatory. IdentityProvider is
	// optional based on whether the client is doing TLS or mTLS.
	if hi.rootProvider == nil {
		return nil, errors.New("xds: CertificateProvider to fetch trusted roots is missing, cannot perform TLS handshake. Please check configuration on the management server")
	}

	// InsecureSkipVerify needs to be set to true because we need to perform
	// custom verification to check the SAN on the received certificate.
	// Currently the Go stdlib does complete verification of the cert (which
	// includes hostname verification) or none. We are forced to go with the
	// latter and perform the normal cert validation ourselves.
	cfg := &tls.Config{
		InsecureSkipVerify: true,
		NextProtos:         []string{"h2"},
	}

	km, err := hi.rootProvider.KeyMaterial(ctx)
	if err != nil {
		return nil, fmt.Errorf("xds: fetching trusted roots from CertificateProvider failed: %v", err)
	}
	cfg.RootCAs = km.Roots

	// If AutoHostSNI is true, and the endpoint hostname is present, we use the
	// endpoint hostname as the SNI value and also for SAN validation.
	// Otherwise, we use the SNI value from HandshakeInfo (which is configured
	// by the control plane) and validating SANs based on that.
	sni := hi.sni
	if hi.useAutoHostSNI && hostname != "" {
		sni = hostname
	}

	cfg.VerifyPeerCertificate = hi.buildVerifyFunc(km, true, sni)

	if hi.identityProvider != nil {
		km, err := hi.identityProvider.KeyMaterial(ctx)
		if err != nil {
			return nil, fmt.Errorf("xds: fetching identity certificates from CertificateProvider failed: %v", err)

View on GitHub (pinned to 0c51461d27)