{"record":{"id":"db4e64e648226d41","repo":"hashicorp/nomad","slug":"plugin-is-shut-down","errorCode":null,"errorMessage":"plugin is shut down","messagePattern":"plugin is shut down","errorType":"exception","errorClass":"ErrPluginShutdown","httpStatus":null,"severity":"error","filePath":"plugins/base/structs/errors.go","lineNumber":14,"sourceCode":"// Copyright IBM Corp. 2015, 2026\n// SPDX-License-Identifier: MPL-2.0\n\npackage structs\n\nimport \"errors\"\n\nconst (\n\terrPluginShutdown = \"plugin is shut down\"\n)\n\nvar (\n\t// ErrPluginShutdown is returned when the plugin has shutdown.\n\tErrPluginShutdown = errors.New(errPluginShutdown)\n)\n","sourceCodeStart":1,"sourceCodeEnd":16,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/base/structs/errors.go#L1-L16","documentation":"ErrPluginShutdown is the sentinel error returned by Nomad plugin clients (Exec, Stats, Prestart, fingerprint, collectStats) once the underlying plugin process (e.g. a task driver like exec/docker or a CSI plugin) has terminated. It signals that the gRPC connection to the plugin is gone, so no further calls can succeed. Callers like taskrunner's lazy_handle treat a first occurrence as a trigger to rebuild or reattach to the actual driver handle.","triggerScenarios":"A driver/plugin subprocess crashed, was killed (OOM, node reboot), or exited during a task's lifecycle; calling Exec or Stats on a TaskHandle whose plugin has already exited; retry helper callStatsWithRetry hitting a dead plugin after exhausting reconnect attempts.","commonSituations":"Driver crash mid-task (e.g. exec driver killed by cgroup OOM); Nomad client restart races where the old handle is still referenced; plugin binary failing to launch on the node; CSI plugin container stopped while volumes are still mounted.","solutions":["Check the task/driver logs to find why the plugin process died (OOM, panic, missing binary) and fix the root cause","Let the taskrunner recover: the ErrPluginShutdown-first pattern rebuilds the handle — ensure the driver is healthy so reattach succeeds","Restart the Nomad client agent or the affected allocation if the handle is permanently stale","Upgrade the driver/plugin if crashes are reproducible"],"exampleFix":"out, err := h.Stats(ctx, interval)\nif errors.Is(err, bstructs.ErrPluginShutdown) {\n    // plugin died: fall back to rebuilding the real handle\n    return nil, ErrTaskNotRunning\n}","handlingStrategy":"type-guard","validationCode":null,"typeGuard":"func isPluginShutdown(err error) bool {\n    return errors.Is(err, bstructs.ErrPluginShutdown)\n}","tryCatchPattern":"out, c, err := h.Exec(timeout, cmd, args)\nif errors.Is(err, bstructs.ErrPluginShutdown) {\n    // first occurrence: rebuild handle / report task dead\n    return nil, ErrTaskNotRunning\n}","preventionTips":["Compare against the exported ErrPluginShutdown sentinel with errors.Is, not string matching","Monitor plugin/driver process health (OOM kills, panics)","Rebuild the task handle on first ErrPluginShutdown, as lazy_handle does","Investigate driver logs to fix the crash root cause instead of masking it"],"tags":["nomad","plugin","driver","process-shutdown"],"backgroundTag":"plugin-process-shutdown","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}