hashicorp/nomad · error

failed to read device manager plugin state: %v

Error message

failed to read device manager plugin state: %v

What it means

Returned during restore of the Nomad client state database when the device manager's plugin-state bucket cannot be read from BoltDB. This wraps the underlying deserialization/IO error returned by boltdd.Get and only fires for errors other than a simple key-not-found (which is handled by resetting ps to nil).

Source

Thrown at client/state/db_bolt.go:869

}

// GetDevicePluginState stores the device manager's plugin state or returns an
// error.
func (s *BoltStateDB) GetDevicePluginState() (*dmstate.PluginState, error) {
	var ps *dmstate.PluginState

	err := s.db.View(func(tx *boltdd.Tx) error {
		devBkt := tx.Bucket(devManagerBucket)
		if devBkt == nil {
			// No state, return
			return nil
		}

		// Restore Plugin State if it exists
		ps = &dmstate.PluginState{}
		if err := devBkt.Get(managerPluginStateKey, ps); err != nil {
			if !boltdd.IsErrNotFound(err) {
				return fmt.Errorf("failed to read device manager plugin state: %v", err)
			}

			// Key not found, reset ps to nil
			ps = nil
		}

		return nil
	})

	if err != nil {
		return nil, err
	}

	return ps, nil
}

// PutDriverPluginState stores the driver manager's plugin state or returns an
// error.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v error: if it is a decode error, the state.db is corrupt — stop the Nomad agent and back up the state directory.
2. Restore state.db from state.db.backup (created during upgrades) or from your backups.
3. If no backup exists, move state.db aside and restart the agent; the client will re-register and rebuild device plugin state.
4. Check disk health and free space; run a filesystem check if I/O errors are reported.

Example fix

// before (corrupt state.db blocks startup)
agent start -> failed to read device manager plugin state: decoding failed
// after
systemctl stop nomad
mv /var/lib/nomad/state.db /var/lib/nomad/state.db.corrupt
systemctl start nomad # client re-initializes state

// before starting agent: sanity-check state db
import bolt "go.etcd.io/bbolt"
db, err := bolt.Open(stateDBPath, 0o600, &bolt.Options{ReadOnly: true})
if err != nil { /* restore from state.db.backup */ }

func isNotFound(err error) bool { return errors.Is(err, boltdd.ErrNotFound) }

if err := restore(...); err != nil {
    logger.Error("state restore failed", "error", err)
    // failover: archive corrupt db and reinitialize
    os.Rename(stateDBPath, stateDBPath+".corrupt")
}

When it happens

Trigger: boltdd.Tx.Get on the device-manager bucket with managerPluginStateKey fails with an error other than boltdd.ErrNotFound — e.g. corrupted bytes that fail msgpack unmarshal into dmstate.PluginState, or a bolt I/O error reading the page.

Common situations: Corrupt or truncated state.db after a hard crash/power loss, disk errors, or a state.db written by an incompatible Nomad version so the stored PluginState payload no longer deserializes.

Related errors

• failed to read driver manager plugin state: %v
• failed to read dynamic plugin registry state: %v
• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/775b45f8298ed57a. Report an issue: GitHub.