What causes "plugin group already shutdown" in hashicorp/nomad?

Do not call RegisterAndRun after Shutdown; check the group lifecycle ordering in client setup code

plugin group already shutdown

Error message

plugin group already shutdown

What it means

PluginGroup.RegisterAndRun registers a plugin manager and starts it in a goroutine. If the group has already been shut down (shutdown flag set), registration is refused with this error to prevent adding managers to a stopped group.

Source

Thrown at client/pluginmanager/group.go:43

	mLock sync.Mutex

	logger log.Logger
}

// New returns an initialized PluginGroup
func New(logger log.Logger) *PluginGroup {
	return &PluginGroup{
		managers: []PluginManager{},
		logger:   logger.Named("plugin"),
	}
}

// RegisterAndRun registers the manager and starts it in a separate goroutine
func (m *PluginGroup) RegisterAndRun(manager PluginManager) error {
	m.mLock.Lock()
	defer m.mLock.Unlock()
	if m.shutdown {
		return fmt.Errorf("plugin group already shutdown")
	}
	m.managers = append(m.managers, manager)

	m.logger.Info("starting plugin manager", "plugin-type", manager.PluginType())
	manager.Run()
	return nil
}

// WaitForFirstFingerprint returns a channel which will be closed once all
// plugin managers are ready. A timeout for waiting on each manager is given
func (m *PluginGroup) WaitForFirstFingerprint(ctx context.Context) (<-chan struct{}, error) {
	m.mLock.Lock()
	defer m.mLock.Unlock()
	if m.shutdown {
		return nil, fmt.Errorf("plugin group already shutdown")
	}

	var wg sync.WaitGroup

View on GitHub (pinned to 482b49bf1a)

Solutions

Do not call RegisterAndRun after Shutdown; check the group lifecycle ordering in client setup code
Guard registration with the group's shutdown state or a lifecycle lock in custom integrations
If hit inside Nomad client startup after a shutdown, recreate the PluginGroup/client instead of reusing the old one
Retry client construction (NewClient) once the previous client has fully exited

Example fix

// before
client.shutdownCh <- struct{}{}
client.pluginManager.RegisterAndRun(driverMgr)  // panics-ish: returns already shutdown
// after
if !client.exited() {
    if err := client.pluginManager.RegisterAndRun(driverMgr); err != nil {
        logger.Error("plugin group unavailable", "err", err)
    }
}

Defensive patterns

Strategy: try-catch

Try / catch

if err := group.RegisterAndRun(manager); err != nil {
    if err.Error() == "plugin group already shutdown" {
        // recreate the PluginGroup/client instead of retrying registration
        group = pluginmanager.NewPluginGroup(logger)
        return group.RegisterAndRun(manager)
    }
    return err
}

Prevention

Serialize client lifecycle: never register managers after calling Shutdown
Use a done/shutdown flag in custom integrations to gate RegisterAndRun calls
Rebuild PluginGroup objects on client reload instead of reusing shut-down ones
Add a mutex or single-flight wrapper around client construction and shutdown

When it happens

Trigger: RegisterAndRun (or BuildPluginGroup usage) is invoked after PluginGroup.Shutdown() has completed — e.g. a late client init path or NewClient retry trying to register a CSI/driver manager onto a shutting-down group.

Common situations: Nomad client shutdown racing with agent reinitialization; custom code or tests constructing a client concurrently with shutdown; calling RegisterAndRun twice around a lifecycle where Shutdown already ran (e.g. reloading client config).

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/55a50f7de30c293e. Report an issue: GitHub.