hashicorp/nomad · error · ErrPluginShutdown

plugin is shut down

Error message

plugin is shut down

What it means

ErrPluginShutdown is the sentinel error returned by Nomad plugin clients (Exec, Stats, Prestart, fingerprint, collectStats) once the underlying plugin process (e.g. a task driver like exec/docker or a CSI plugin) has terminated. It signals that the gRPC connection to the plugin is gone, so no further calls can succeed. Callers like taskrunner's lazy_handle treat a first occurrence as a trigger to rebuild or reattach to the actual driver handle.

Source

Thrown at plugins/base/structs/errors.go:14

// Copyright IBM Corp. 2015, 2026
// SPDX-License-Identifier: MPL-2.0

package structs

import "errors"

const (
	errPluginShutdown = "plugin is shut down"
)

var (
	// ErrPluginShutdown is returned when the plugin has shutdown.
	ErrPluginShutdown = errors.New(errPluginShutdown)
)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the task/driver logs to find why the plugin process died (OOM, panic, missing binary) and fix the root cause
2. Let the taskrunner recover: the ErrPluginShutdown-first pattern rebuilds the handle — ensure the driver is healthy so reattach succeeds
3. Restart the Nomad client agent or the affected allocation if the handle is permanently stale
4. Upgrade the driver/plugin if crashes are reproducible

Example fix

out, err := h.Stats(ctx, interval)
if errors.Is(err, bstructs.ErrPluginShutdown) {
    // plugin died: fall back to rebuilding the real handle
    return nil, ErrTaskNotRunning
}

func isPluginShutdown(err error) bool {
    return errors.Is(err, bstructs.ErrPluginShutdown)
}

out, c, err := h.Exec(timeout, cmd, args)
if errors.Is(err, bstructs.ErrPluginShutdown) {
    // first occurrence: rebuild handle / report task dead
    return nil, ErrTaskNotRunning
}

When it happens

Trigger: A driver/plugin subprocess crashed, was killed (OOM, node reboot), or exited during a task's lifecycle; calling Exec or Stats on a TaskHandle whose plugin has already exited; retry helper callStatsWithRetry hitting a dead plugin after exhausting reconnect attempts.

Common situations: Driver crash mid-task (e.g. exec driver killed by cgroup OOM); Nomad client restart races where the old handle is still referenced; plugin binary failing to launch on the node; CSI plugin container stopped while volumes are still mounted.

Related errors

• could not validate task driver capabilities: %v
• failed to start plugin: %v
• plugin loaded does not implement the driver interface
• init of plugin %s failed: %w
• ErrDriverNotFound

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/db4e64e648226d41. Report an issue: GitHub.