hashicorp/nomad · error

failed to read plugin state: %v

Error message

failed to read plugin state: %v

What it means

cleanupStalePlugins (run at manager startup from Run) reads the persisted device plugin state via state.GetDevicePluginState. If reading state fails, the error is wrapped. The manager cannot determine which plugins were previously launched, so startup cleanup is aborted.

Source

Thrown at client/devicemanager/manager.go:276

		if !i.HasDevices(d) {
			continue
		}

		// We found a match so reserve
		return i.DeviceStats(d), nil
	}

	return nil, UnknownDeviceErrFromAllocated("failed to collect statistics", d)
}

// cleanupStalePlugins reads the device managers state and shuts down any
// previously launched plugin.
func (m *manager) cleanupStalePlugins() error {

	// Read the old plugin state
	s, err := m.state.GetDevicePluginState()
	if err != nil {
		return fmt.Errorf("failed to read plugin state: %v", err)
	}

	// No state was stored so there is nothing to do.
	if s == nil {
		return nil
	}

	// For each plugin go through and try to shut it down
	var mErr multierror.Error
	for name, c := range s.ReattachConfigs {
		rc, err := pstructs.ReattachConfigToGoPlugin(c)
		if err != nil {
			_ = multierror.Append(&mErr, fmt.Errorf("failed to convert reattach config: %v", err))
			continue
		}

		instance, err := m.loader.Reattach(name, base.PluginTypeDevice, rc)
		if err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the underlying error in client logs and fix disk/permission issues on data_dir.
2. Restart the nomad client after repairing the state store; bolt may recover a cleanly closed DB.
3. As a last resort (data loss acceptable), back up and remove the corrupted client state DB so a fresh one is created, then restart.
4. Ensure data_dir is on healthy writable storage before restarting.

Example fix

// before: state db corrupt, agent crash-loops
# check disk and perms
sudo chown -R nomad:nomad /var/lib/nomad && ls -ld /var/lib/nomad/data
// after: repair or reset state
mv /var/lib/nomad/data /var/lib/nomad/data.bak && systemctl restart nomad

// health check state dir at startup
if fi, err := os.Stat(dataDir); err != nil || !fi.IsDir() { fixPermsOrCreate(dataDir) }

if err := manager.Run(); err != nil && strings.Contains(err.Error(), "failed to read plugin state") {
  // alert, repair data_dir/state.db, then restart client
  log.Errorf("device state unreadable: %v", err)
}

When it happens

Trigger: Nomad client startup with a corrupted/unreadable bolt state DB or an I/O error while the state store reads the device plugin reattach-config bucket.

Common situations: Disk full or permissions problem on the client data_dir; corrupted state.db after crash; state directory restored from another node.

Related errors

• failed to convert reattach config: %v
• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state
• rpc.dial_timeout must be greater than or equal to zero

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/2d16107ffd70d673. Report an issue: GitHub.