{"record":{"id":"e25aa8ec8ef048d1","repo":"grpc/grpc-go","slug":"xds-failed-to-get-security-plugin-instance-v","errorCode":null,"errorMessage":"xds: failed to get security plugin instance (%+v): %v","messagePattern":"xds: failed to get security plugin instance \\(%\\+v\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/xds/balancer/clusterimpl/clusterimpl.go","lineNumber":323,"sourceCode":"\t\tb.loadWrapper.UpdateLoadStore(loadStore)\n\t}\n\n\treturn nil\n}\n\nfunc buildProviderFunc(configs map[string]*certprovider.BuildableConfig, instanceName, certName string, wantIdentity, wantRoot bool) (certprovider.Provider, error) {\n\tcfg := configs[instanceName]\n\tprovider, err := cfg.Build(certprovider.BuildOptions{\n\t\tCertName:     certName,\n\t\tWantIdentity: wantIdentity,\n\t\tWantRoot:     wantRoot,\n\t})\n\tif err != nil {\n\t\t// This error is not expected since the bootstrap process parses the\n\t\t// config and makes sure that it is acceptable to the plugin. Still, it\n\t\t// is possible that the plugin parses the config successfully, but its\n\t\t// Build() method errors out.\n\t\treturn nil, fmt.Errorf(\"xds: failed to get security plugin instance (%+v): %v\", cfg, err)\n\t}\n\treturn provider, nil\n}\n\nfunc (b *clusterImplBalancer) buildProviders(config *xdsresource.SecurityConfig) (certprovider.Provider, certprovider.Provider, error) {\n\tcpc := b.xdsClient.BootstrapConfig().CertProviderConfigs()\n\tvar rootProvider certprovider.Provider\n\tif config.UseSystemRootCerts {\n\t\trootProvider = systemRootCertsProvider{}\n\t} else {\n\t\trp, err := buildProvider(cpc, config.RootInstanceName, config.RootCertName, false, true)\n\t\tif err != nil {\n\t\t\treturn nil, nil, err\n\t\t}\n\t\trootProvider = rp\n\t}\n\n\tvar identityProvider certprovider.Provider","sourceCodeStart":305,"sourceCodeEnd":341,"githubUrl":"https://github.com/grpc/grpc-go/blob/0c51461d27177d997e14c642fe18c11668fc09a3/internal/xds/balancer/clusterimpl/clusterimpl.go#L305-L341","documentation":"Returned by buildProviderFunc (clusterimpl.go:323) when cfg.Build() fails for a certificate provider plugin during xDS security config processing. The %+v shows the full provider BuildableConfig struct (for diagnostics) and %v shows the Build error. The comment (lines 319-322) notes this is unexpected because bootstrap already parsed and validated the config, but Build() can still fail at instantiation time (e.g., file not found, plugin unavailable).","triggerScenarios":"handleSecurityConfig (line 359) is called during a cluster_impl UpdateClientConnState when the cluster resource contains a SecurityConfig. It calls buildProviders (line 328) which calls buildProviderFunc for root and/or identity certificate providers. cfg.Build() fails because the certificate provider plugin (e.g., file-based cert provider) cannot initialize — missing cert files, wrong paths, permission denied, or a custom plugin that errors on startup.","commonSituations":"The bootstrap configuration references certificate files that don't exist at the specified path. The certificate provider plugin (e.g., Google S2A, FileWatcher) fails to initialize because its specific config is wrong (wrong cert file format, key mismatch). Permissions issue reading the cert/key files. The bootstrap JSON was generated for a different environment (e.g., dev bootstrap used in prod). A custom cert provider plugin that returns an error from Build().","solutions":["Inspect the %+v to see which certificate provider instance name and config is failing.","Verify the certificate and key files referenced in the bootstrap config exist and are readable by the process: 'ls -la /path/to/cert' and check file permissions.","Validate the bootstrap JSON: ensure the certificate_provider_instances section has correct plugin configs with valid file paths.","If using a custom cert provider plugin, debug its Build() method for the specific error.","Confirm the cert and key files are valid PEM/DER and the key matches the certificate.","Check if the bootstrap config's credential_instance_name matches what the xDS server's security config references."],"exampleFix":"// before: bootstrap references missing cert files\n{\n  \"certificate_provider_instances\": {\n    \"default\": {\n      \"plugin_name\": \"file_watcher\",\n      \"config\": {\n        \"certificate_file\": \"/etc/certs/server.crt\",\n        \"private_key_file\": \"/etc/certs/server.key\"\n      }\n    }\n  }\n}\n// certs don't exist at those paths\n\n// after: ensure files exist and are readable, or fix paths\n// ls -la /etc/certs/server.crt /etc/certs/server.key\n// chmod 644 /etc/certs/server.crt && chmod 600 /etc/certs/server.key\n// or update bootstrap JSON with correct paths before starting the client","handlingStrategy":"validation","validationCode":"// Validate certificate provider config before the channel uses it.\n// Check that referenced files exist and are readable:\nfunc validateCertProviderFiles(bootstrapPath string) error {\n    data, err := os.ReadFile(bootstrapPath)\n    if err != nil {\n        return fmt.Errorf(\"cannot read bootstrap: %w\", err)\n    }\n    // Parse and check file_watcher plugin paths\n    var bs struct {\n        CertProviderInstances map[string]struct {\n            PluginName string `json:\"plugin_name\"`\n            Config     struct {\n                CertFile     string `json:\"certificate_file\"`\n                KeyFile      string `json:\"private_key_file\"`\n                RootCertFile string `json:\"root_certificate_file\"`\n            } `json:\"config\"`\n        } `json:\"certificate_provider_instances\"`\n    }\n    json.Unmarshal(data, &bs)\n    for name, inst := range bs.CertProviderInstances {\n        if inst.PluginName == \"file_watcher\" {\n            for _, f := range []string{inst.Config.CertFile, inst.Config.KeyFile, inst.Config.RootCertFile} {\n                if f != \"\" {\n                    if _, err := os.Stat(f); err != nil {\n                        return fmt.Errorf(\"instance %q: file %q: %w\", name, f, err)\n                    }\n                }\n            }\n        }\n    }\n    return nil\n}","typeGuard":null,"tryCatchPattern":"// This error propagates from cluster_impl's handleSecurityConfig\n// through UpdateClientConnState. The channel enters TRANSIENT_FAILURE.\nerr := getChannelError(conn)\nif err != nil && strings.Contains(err.Error(), \"failed to get security plugin\") {\n    // inspect the %+v in the message for the provider instance name\n    // check certificate files and bootstrap config\n    log.Printf(\"cert provider build failed: %v\", err)\n}","preventionTips":["Validate the bootstrap configuration's certificate provider section before starting the client.","Ensure certificate and key files exist, are valid PEM, and are readable by the process.","Use a file watcher or cert manager to ensure certs are present and rotated.","Test the bootstrap config in staging with the same cert layout as production.","Verify the identity/root instance names in the security config match the bootstrap's certificate_provider_instances keys."],"tags":["xds","security","tls","certificate-provider","bootstrap","cluster-impl"],"backgroundTag":null,"analyzedSha":"0c51461d27177d997e14c642fe18c11668fc09a3","analyzedAt":"2026-08-11T14:49:15.055Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}