hashicorp/nomad · error

failed to read dynamic plugin registry state: %v

Error message

failed to read dynamic plugin registry state: %v

What it means

Returned during state restore when the dynamic plugin registry (CSI/host-volume plugin) state cannot be read from its bucket. Like the other plugin-state errors, a boltdd not-found is tolerated (registry state reset to nil); any other read/decode failure is wrapped and aborts restore.

Source

Thrown at client/state/db_bolt.go:962

}

// GetDynamicPluginRegistryState stores the dynamic plugin registry's
// registry state or returns an error.
func (s *BoltStateDB) GetDynamicPluginRegistryState() (*dynamicplugins.RegistryState, error) {
	var ps *dynamicplugins.RegistryState

	err := s.db.View(func(tx *boltdd.Tx) error {
		dynamicBkt := tx.Bucket(dynamicPluginBucketName)
		if dynamicBkt == nil {
			// No state, return
			return nil
		}

		// Restore Plugin State if it exists
		ps = &dynamicplugins.RegistryState{}
		if err := dynamicBkt.Get(registryStateKey, ps); err != nil {
			if !boltdd.IsErrNotFound(err) {
				return fmt.Errorf("failed to read dynamic plugin registry state: %v", err)
			}

			// Key not found, reset ps to nil
			ps = nil
		}

		return nil
	})

	if err != nil {
		return nil, err
	}

	return ps, nil
}

func keyForCheck(allocID string, checkID structs.CheckID) []byte {
	return fmt.Appendf([]byte{}, "%s_%s", allocID, checkID)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped error to confirm decode vs I/O failure.
2. Restore state.db from backup; otherwise move it aside and let the client rebuild registry state (CSI plugins re-register and volumes re-detect).
3. Check storage health on the node before restarting.
4. Avoid downgrading Nomad on nodes sharing a state directory.

Example fix

// before
failed to read dynamic plugin registry state: decoding failed
// after
systemctl stop nomad
mv /var/lib/nomad/state.db /var/lib/nomad/state.db.corrupt
systemctl start nomad

// preflight: attempt a read of the registry state key
db.View(func(tx *boltdd.Tx) error {
    ps := &dynamicplugins.RegistryState{}
    return tx.Bucket([]byte(dynamicBktName)).Get(registryStateKey, ps)
})

func isNotFound(err error) bool { return errors.Is(err, boltdd.ErrNotFound) }

if err := restoreDynamicRegistry(); err != nil && !errors.Is(err, boltdd.ErrNotFound) {
    logger.Warn("dropping dynamic registry state; plugins will re-register", "error", err)
    resetStateDB(stateDir)
}

When it happens

Trigger: dynamicBkt.Get(registryStateKey) fails decoding into dynamicplugins.RegistryState or hits a bolt I/O error; typically with a corrupt state.db or payload from an incompatible schema.

Common situations: Crash during a CSI plugin state write, Nomad version mismatch in the client data dir, or underlying disk corruption on the node.

Related errors

• failed to read device manager plugin state: %v
• failed to read driver manager plugin state: %v
• error parsing: root should be an object
• cannot specify Accessor ID
• error getting plugin: %s, %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1a91ae8fa3ce05dd. Report an issue: GitHub.