hashicorp/nomad · warning

singleton plugin exited

Error message

singleton plugin exited

What it means

SingletonPluginExited is a sentinel error returned by the SingletonLoader's dispense/getPlugin when the currently loaded singleton plugin process has exited. It is not fatal: callers are expected to compare with it and retry dispense, which launches a new plugin instance.

Source

Thrown at helper/pluginutils/singleton/singleton.go:20

// SPDX-License-Identifier: BUSL-1.1

package singleton

import (
	"fmt"
	"sync"

	log "github.com/hashicorp/go-hclog"
	plugin "github.com/hashicorp/go-plugin"
	"github.com/hashicorp/nomad/helper/pluginutils/loader"
	"github.com/hashicorp/nomad/plugins/base"
)

var (
	// SingletonPluginExited is returned when the dispense is called and the
	// existing plugin has exited. The caller should retry, and this will issue
	// a new plugin instance.
	SingletonPluginExited = fmt.Errorf("singleton plugin exited")
)

// SingletonLoader is used to only load a single external plugin at a time.
type SingletonLoader struct {
	// Loader is the underlying plugin loader that we wrap to give a singleton
	// behavior.
	loader loader.PluginCatalog

	// instances is a mapping of the plugin to a future which holds a plugin
	// instance
	instances    map[loader.PluginID]*future
	instanceLock sync.Mutex

	// logger is the logger used by the singleton
	logger log.Logger
}

// NewSingletonLoader wraps a plugin catalog and provides singleton behavior on

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the dispense (the library contract) — callers like device/driver managers already do this automatically
2. Check why the plugin exited: inspect its stderr/stdout logs for panics or OOM kills
3. Upgrade or fix the plugin binary that keeps dying
4. Ensure sufficient memory/resources so the plugin process is not killed

Example fix

// before
pluginInstance, err = i.loader.Dispense(...)
if err != nil { return err }
// after
if err == singleton.SingletonPluginExited {
    pluginInstance, err = i.loader.Dispense(...) // retry spawns new instance
}

inst, err := i.loader.Dispense(...)
if err != nil {
    if errors.Is(err, singleton.SingletonPluginExited) {
        inst, err = i.loader.Dispense(...) // one retry launches a new instance
    }
    if err != nil { return err }
}

When it happens

Trigger: Dispense through a SingletonLoader when the cached plugin process died; seen in client/devicemanager/instance.go:284 and client/pluginmanager/drivermanager/instance.go:213.

Common situations: Driver/device plugin binary crashing (OOM, segfault), plugin upgraded on disk while running, or agent-side kill of the plugin process.

Related errors

• retry config is nil or empty
• CSI client error (retryable)
• %w: destroying network to retry failed: %v
• %w: %v; see: <https://developer.hashicorp.com/nomad/s/envoy-
• missing %q

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6aac1b0ea858f3cf. Report an issue: GitHub.