grpc/grpc-go · error

xds: failed to get security plugin instance (%+v): %v

Error message

xds: failed to get security plugin instance (%+v): %v

What it means

The cluster_impl balancer builds certificate provider plugins from the xDS bootstrap configuration's security settings. This error fires when a certificate provider's Build() method fails at runtime, even though the bootstrap config parsed successfully. The provider could be for identity certs (mTLS client), root certs (CA verification), or both.

Source

Thrown at internal/xds/balancer/clusterimpl/clusterimpl.go:323

		b.loadWrapper.UpdateLoadStore(loadStore)
	}

	return nil
}

func buildProviderFunc(configs map[string]*certprovider.BuildableConfig, instanceName, certName string, wantIdentity, wantRoot bool) (certprovider.Provider, error) {
	cfg := configs[instanceName]
	provider, err := cfg.Build(certprovider.BuildOptions{
		CertName:     certName,
		WantIdentity: wantIdentity,
		WantRoot:     wantRoot,
	})
	if err != nil {
		// This error is not expected since the bootstrap process parses the
		// config and makes sure that it is acceptable to the plugin. Still, it
		// is possible that the plugin parses the config successfully, but its
		// Build() method errors out.
		return nil, fmt.Errorf("xds: failed to get security plugin instance (%+v): %v", cfg, err)
	}
	return provider, nil
}

func (b *clusterImplBalancer) buildProviders(config *xdsresource.SecurityConfig) (certprovider.Provider, certprovider.Provider, error) {
	cpc := b.xdsClient.BootstrapConfig().CertProviderConfigs()
	var rootProvider certprovider.Provider
	if config.UseSystemRootCerts {
		rootProvider = systemRootCertsProvider{}
	} else {
		rp, err := buildProvider(cpc, config.RootInstanceName, config.RootCertName, false, true)
		if err != nil {
			return nil, nil, err
		}
		rootProvider = rp
	}

	var identityProvider certprovider.Provider

View on GitHub (pinned to 03255a9237)

Solutions

1. Verify the certificate and key files referenced in the xDS bootstrap config exist and are readable by the process
2. Check that the certificate and private key match (openssl x509 -noout -modulus | openssl md5 vs openssl rsa -noout -modulus | openssl md5)
3. Inspect the xDS bootstrap config's certificate_providers section for correct instance names, file paths, and plugin types
4. If running in Kubernetes, verify ConfigMaps/Secrets are mounted correctly at the expected paths
5. Check the wrapped error (the %v part) for the specific provider error message

Example fix

// before: bootstrap references non-existent cert files
{"certificate_providers": {"default": {"plugin_name": "file_watcher", "config": {"certificate_file": "/etc/certs/missing.crt", "private_key_file": "/etc/certs/server.key"}}}}
// after: correct file paths
{"certificate_providers": {"default": {"plugin_name": "file_watcher", "config": {"certificate_file": "/etc/certs/tls.crt", "private_key_file": "/etc/certs/tls.key"}}}}

// Verify certificate files exist before starting with xDS security
func validateCertFiles(bootstrapPath string) error {
    data, err := os.ReadFile(bootstrapPath)
    if err != nil { return err }
    var cfg struct {
        CertProviders map[string]struct {
            PluginName string `json:"plugin_name"`
            Config     struct {
                CertFile     string `json:"certificate_file"`
                PrivateKeyFile string `json:"private_key_file"`
                RootFile     string `json:"root_certificate_file"`
            } `json:"config"`
        } `json:"certificate_providers"`
    }
    json.Unmarshal(data, &cfg)
    for name, p := range cfg.CertProviders {
        for _, f := range []string{p.Config.CertFile, p.Config.PrivateKeyFile, p.Config.RootFile} {
            if f != "" {
                if _, err := os.Stat(f); err != nil {
                    return fmt.Errorf("provider %s: %s: %w", name, f, err)
                }
            }
        }
    }
    return nil
}

// Monitor channel connectivity after xDS security config updates
if conn.GetState() == connectivity.TransientFailure {
    // check logs for 'failed to get security plugin instance'
    // verify cert files and bootstrap provider configs
}

When it happens

Trigger: Triggered in buildProviderFunc() when cfg.Build() returns an error. The cfg is a certprovider.BuildableConfig looked up from b.xdsClient.BootstrapConfig().CertProviderConfigs() by instance name. Build() can fail if the referenced certificate file doesn't exist, the private key doesn't match the certificate, or the provider plugin has runtime initialization errors.

Common situations: The certificate file or key referenced in the xDS bootstrap config doesn't exist at runtime (wrong path, volume not mounted in Kubernetes); the TLS certificate and private key don't match; the certificate provider plugin (e.g., file-based, or a custom plugin) has a bug in its Build() method; the bootstrap config was generated with a different environment than where it runs.

Related errors

• received Cluster resource that contains invalid security con
• UpstreamTlsContext in CDS response does not contain a Common
• security configuration on the server-side does not contain i
• security configuration on the client-side does not contain r
• DownstreamTlsContext in LDS response does not contain a Comm

AI-assisted analysis of grpc/grpc-go@03255a9237 (2026-08-07). Data as JSON: /api/errors/e25aa8ec8ef048d1. Report an issue: GitHub.