{"record":{"id":"de145733d870062a","repo":"hashicorp/nomad","slug":"failed-executing-plugin-q-for-secret-q-w","errorCode":null,"errorMessage":"failed executing plugin %q for secret %q: %w","messagePattern":"failed executing plugin %q for secret %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/secrets/plugin_provider.go","lineNumber":55,"sourceCode":"\treturn &ExternalPluginProvider{\n\t\tplugin:     plugin,\n\t\tpluginName: pluginName,\n\t\tsecretName: secretName,\n\t\tpath:       path,\n\t\tenv:        env,\n\t}\n}\n\nfunc (p *ExternalPluginProvider) InterpolateEnv(interpolate func(string) string) {\n\tfor key, value := range p.env {\n\t\tp.env[key] = interpolate(value)\n\t}\n}\n\nfunc (p *ExternalPluginProvider) Fetch(ctx context.Context) (map[string]string, error) {\n\tresp, err := p.plugin.Fetch(ctx, p.path, p.env)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed executing plugin %q for secret %q: %w\", p.pluginName, p.secretName, err)\n\t}\n\tif resp.Error != nil {\n\t\treturn nil, fmt.Errorf(\"provider %q for secret %q response contained error: %q\", p.pluginName, p.secretName, *resp.Error)\n\t}\n\n\tformatted := make(map[string]string, len(resp.Result))\n\tfor k, v := range resp.Result {\n\t\tformatted[fmt.Sprintf(\"secret.%s.%s\", p.secretName, k)] = v\n\t}\n\n\treturn formatted, nil\n}\n","sourceCodeStart":37,"sourceCodeEnd":68,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/secrets/plugin_provider.go#L37-L68","documentation":"This error wraps any failure from the external secret plugin's Fetch call inside Nomad's ExternalPluginProvider.Fetch. It means the plugin binary itself failed (crashed, non-zero exit, protocol error, timeout, or context cancellation) while retrieving the secret, and Nomad wraps the underlying cause with %w so the root error is preserved in the chain.","triggerScenarios":"Calling ExternalPluginProvider.Fetch (via the taskrunner secrets hook) when p.plugin.Fetch returns an error: plugin binary missing/not executable, plugin crashes or exits non-zero, plugin returns a malformed gRPC response, or ctx is cancelled/timed out during the fetch.","commonSituations":"Nomad client agents missing the vault-secrets or custom secret plugin binary in plugin_dir; plugin version incompatible with the Nomad API protocol; plugin lacking exec permissions after packaging; network/credential issues inside the plugin causing it to return an error; task shutdown cancelling the context mid-fetch.","solutions":["Inspect the wrapped root cause (%w) in the error chain — fix whatever the plugin itself reported (credentials, network, config).","Verify the plugin binary exists in the agent's plugin_dir, is executable, and matches the Nomad secrets-plugin protocol version.","Test the plugin standalone with its CLI/dev mode to reproduce and debug the failure.","Check agent logs for plugin lifecycle errors (launch, handshake, protocol mismatch)."],"exampleFix":"// before (plugin returns error due to missing CLOUD_TOKEN env)\nresp, err := plugin.Fetch(ctx, path, env) // err: \"missing CLOUD_TOKEN\"\n// after (ensure required env is passed to the plugin via the task env or agent config)\nenv[\"CLOUD_TOKEN\"] = os.Getenv(\"CLOUD_TOKEN\")\nresp, err := plugin.Fetch(ctx, path, env)","handlingStrategy":"try-catch","validationCode":"// before scheduling the task, verify the plugin is deployable on the client\nif _, err := os.Stat(filepath.Join(pluginDir, pluginName)); err != nil {\n    return fmt.Errorf(\"secret plugin %s not present in plugin_dir: %w\", pluginName, err)\n}\nif err := os.Chmod(filepath.Join(pluginDir, pluginName), 0o755); err != nil { /* not executable */ }","typeGuard":null,"tryCatchPattern":"secrets, err := provider.Fetch(ctx)\nif err != nil {\n    var perr *PluginError\n    if errors.As(err, &perr) {\n        log.Error(\"secret plugin failed\", \"plugin\", pluginName, \"cause\", errors.Unwrap(err))\n        // retry with backoff or fail the task hook with the root cause\n    }\n    return err\n}","preventionTips":["Ship and pin the plugin binary in the agent's plugin_dir on every client via provisioning/config management.","Test plugin upgrades against the Nomad plugin protocol version before rollout.","Give the plugin a deadline (ctx timeout) so hangs surface as clear errors.","Monitor agent logs for plugin launch/handshake failures after client upgrades."],"tags":["nomad","secrets","plugin","grpc"],"backgroundTag":"secret-plugin-execution-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}