grpc/grpc-go · error

xds: failed to get security plugin instance (%+v)

Error message

xds: failed to get security plugin instance (%+v): %v

What it means

Returned by buildProviderFunc (clusterimpl.go:323) when cfg.Build() fails for a certificate provider plugin during xDS security config processing. The %+v shows the full provider BuildableConfig struct (for diagnostics) and %v shows the Build error. The comment (lines 319-322) notes this is unexpected because bootstrap already parsed and validated the config, but Build() can still fail at instantiation time (e.g., file not found, plugin unavailable).

Solutions

  1. Inspect the %+v to see which certificate provider instance name and config is failing.
  2. Verify the certificate and key files referenced in the bootstrap config exist and are readable by the process: 'ls -la /path/to/cert' and check file permissions.
  3. Validate the bootstrap JSON: ensure the certificate_provider_instances section has correct plugin configs with valid file paths.
  4. If using a custom cert provider plugin, debug its Build() method for the specific error.
  5. Confirm the cert and key files are valid PEM/DER and the key matches the certificate.
  6. Check if the bootstrap config's credential_instance_name matches what the xDS server's security config references.

Example fix

// before: bootstrap references missing cert files
{
  "certificate_provider_instances": {
    "default": {
      "plugin_name": "file_watcher",
      "config": {
        "certificate_file": "/etc/certs/server.crt",
        "private_key_file": "/etc/certs/server.key"
      }
    }
  }
}
// certs don't exist at those paths

// after: ensure files exist and are readable, or fix paths
// ls -la /etc/certs/server.crt /etc/certs/server.key
// chmod 644 /etc/certs/server.crt && chmod 600 /etc/certs/server.key
// or update bootstrap JSON with correct paths before starting the client
Defensive patterns

Strategy: validation

Validate before calling

// Validate certificate provider config before the channel uses it.
// Check that referenced files exist and are readable:
func validateCertProviderFiles(bootstrapPath string) error {
    data, err := os.ReadFile(bootstrapPath)
    if err != nil {
        return fmt.Errorf("cannot read bootstrap: %w", err)
    }
    // Parse and check file_watcher plugin paths
    var bs struct {
        CertProviderInstances map[string]struct {
            PluginName string `json:"plugin_name"`
            Config     struct {
                CertFile     string `json:"certificate_file"`
                KeyFile      string `json:"private_key_file"`
                RootCertFile string `json:"root_certificate_file"`
            } `json:"config"`
        } `json:"certificate_provider_instances"`
    }
    json.Unmarshal(data, &bs)
    for name, inst := range bs.CertProviderInstances {
        if inst.PluginName == "file_watcher" {
            for _, f := range []string{inst.Config.CertFile, inst.Config.KeyFile, inst.Config.RootCertFile} {
                if f != "" {
                    if _, err := os.Stat(f); err != nil {
                        return fmt.Errorf("instance %q: file %q: %w", name, f, err)
                    }
                }
            }
        }
    }
    return nil
}

Try / catch

// This error propagates from cluster_impl's handleSecurityConfig
// through UpdateClientConnState. The channel enters TRANSIENT_FAILURE.
err := getChannelError(conn)
if err != nil && strings.Contains(err.Error(), "failed to get security plugin") {
    // inspect the %+v in the message for the provider instance name
    // check certificate files and bootstrap config
    log.Printf("cert provider build failed: %v", err)
}

Prevention

When it happens

Trigger: handleSecurityConfig (line 359) is called during a cluster_impl UpdateClientConnState when the cluster resource contains a SecurityConfig. It calls buildProviders (line 328) which calls buildProviderFunc for root and/or identity certificate providers. cfg.Build() fails because the certificate provider plugin (e.g., file-based cert provider) cannot initialize — missing cert files, wrong paths, permission denied, or a custom plugin that errors on startup.

Common situations: The bootstrap configuration references certificate files that don't exist at the specified path. The certificate provider plugin (e.g., Google S2A, FileWatcher) fails to initialize because its specific config is wrong (wrong cert file format, key mismatch). Permissions issue reading the cert/key files. The bootstrap JSON was generated for a different environment (e.g., dev bootstrap used in prod). A custom cert provider plugin that returns an error from Build().

Related errors

AI-assisted analysis of grpc/grpc-go@0c51461d27 (2026-08-11). Data as JSON: /api/errors/e25aa8ec8ef048d1. Report an issue: GitHub.

Appendix: source

Thrown at internal/xds/balancer/clusterimpl/clusterimpl.go:323

		b.loadWrapper.UpdateLoadStore(loadStore)
	}

	return nil
}

func buildProviderFunc(configs map[string]*certprovider.BuildableConfig, instanceName, certName string, wantIdentity, wantRoot bool) (certprovider.Provider, error) {
	cfg := configs[instanceName]
	provider, err := cfg.Build(certprovider.BuildOptions{
		CertName:     certName,
		WantIdentity: wantIdentity,
		WantRoot:     wantRoot,
	})
	if err != nil {
		// This error is not expected since the bootstrap process parses the
		// config and makes sure that it is acceptable to the plugin. Still, it
		// is possible that the plugin parses the config successfully, but its
		// Build() method errors out.
		return nil, fmt.Errorf("xds: failed to get security plugin instance (%+v): %v", cfg, err)
	}
	return provider, nil
}

func (b *clusterImplBalancer) buildProviders(config *xdsresource.SecurityConfig) (certprovider.Provider, certprovider.Provider, error) {
	cpc := b.xdsClient.BootstrapConfig().CertProviderConfigs()
	var rootProvider certprovider.Provider
	if config.UseSystemRootCerts {
		rootProvider = systemRootCertsProvider{}
	} else {
		rp, err := buildProvider(cpc, config.RootInstanceName, config.RootCertName, false, true)
		if err != nil {
			return nil, nil, err
		}
		rootProvider = rp
	}

	var identityProvider certprovider.Provider

View on GitHub (pinned to 0c51461d27)