hashicorp/nomad · error

failed to read dynamic plugin registry state: %v

Error message

failed to read dynamic plugin registry state: %v

What it means

UpgradeDynamicPluginRegistry reads the dynamic plugin RegistryState12 blob from the client state store. If dynamicBkt.Get fails with anything other than boltdd's not-found error, the upgrade cannot read prior plugin registrations and returns this wrapped error to abort the migration.

Source

Thrown at client/state/upgrade.go:335

		if err := putTaskStateImpl(tx, allocID, taskName, taskState); err != nil {
			return err
		}
	}

	return nil
}

func UpgradeDynamicPluginRegistry(logger hclog.Logger, tx *boltdd.Tx) error {

	dynamicBkt := tx.Bucket(dynamicPluginBucketName)
	if dynamicBkt == nil {
		return nil // no previous plugins upgrade
	}

	oldState := &RegistryState12{}
	if err := dynamicBkt.Get(registryStateKey, oldState); err != nil {
		if !boltdd.IsErrNotFound(err) {
			return fmt.Errorf("failed to read dynamic plugin registry state: %v", err)
		}
	}

	newState := &dynamicplugins.RegistryState{
		Plugins: make(map[string]map[string]*list.List),
	}

	for ptype, plugins := range oldState.Plugins {
		newState.Plugins[ptype] = make(map[string]*list.List)
		for pname, pluginInfo := range plugins {
			newState.Plugins[ptype][pname] = list.New()
			entry := list.Element{Value: pluginInfo}
			newState.Plugins[ptype][pname].PushFront(entry)
		}
	}
	return dynamicBkt.Put(registryStateKey, newState)
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v error: if it's a decode failure, the registry blob is corrupt and should be dropped (tasks re-register plugins at restart)
2. Back up and clean/restore the client state file, then restart nomad so plugins re-discover
3. Avoid skipping version ranges; migrate through the documented upgrade path
4. Report upstream with the wrapped error if it reproduces on healthy state

// Go: pre-check the registry key decodes before migration
var probe RegistryState12
if err := dynamicBkt.Get(registryStateKey, &probe); err != nil && !boltdd.IsErrNotFound(err) {
    return fmt.Errorf("pre-flight: dynamic plugin registry unreadable: %w", err)
}

// Go: distinguish benign absence from real failure
if err := dynamicBkt.Get(registryStateKey, oldState); err != nil {
    if boltdd.IsErrNotFound(err) {
        return nil // fresh state, nothing to upgrade
    }
    return fmt.Errorf("failed to read dynamic plugin registry state: %v", err)
}

// Go
if err := UpgradeDynamicPluginRegistry(logger, tx); err != nil {
    logger.Error("dynamic plugin registry migration failed; plugins will re-register after state reset", "err", err)
    return err
}

When it happens

Trigger: tx.Bucket(dynamicPluginBucketName) exists but Get(registryStateKey, &RegistryState12{}) fails with a non-NotFound error (msgpack decode failure of the stored blob, or an internal boltdd/bbolt read error).

Common situations: Corrupted dynamic plugin registry state after an unclean shutdown on a client using CSI/docker plugin tasks; schema mismatch when jumping across unsupported Nomad versions; disk errors.

Related errors

• error deleting invalid task state for task %q: %v
• error deleting unexpected task bucket %q: %v
• error delting unexpected task key %q: %v
• task state entry not found
• Plugin.Type must not be empty

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4fb5bad6ee80943f. Report an issue: GitHub.