hashicorp/nomad · error

plugin lookup error

Error message

plugin lookup error: %s %v

What it means

csiVolumeDenormalizePluginsTxn enriches a CSIVolume with health/schedulability data from its plugin. When CSIPluginByIDTxn returns a non-nil error (a memdb lookup failure on the csi_plugins table, not merely a missing plugin), this wrapper propagates it prefixed with the volume's PluginID. Note a nil plugin with nil error is handled gracefully (health zeroed), so this error only fires on genuine lookup failures.

Solutions

1. Restart the server agent to rebuild memdb state from Raft.
2. Check the wrapped inner error (%v) for the actual memdb failure and address it.
3. Ensure the plugin row exists via nomad plugin status / CSIPluginByID; re-register the plugin if needed.
4. Align Nomad versions across the cluster.

// before denormalizing, confirm plugin exists:
plug, err := store.CSIPluginByID(ws, ns, vol.PluginID)
if err != nil { return nil, err }
if plug == nil { vol.Schedulable = false } // missing plugin is not an error

func pluginUsable(p *structs.CSIPlugin) bool {
    return p != nil && p.ControllersHealthy > 0
}

vol, err := store.CSIVolumeByID(ws, ns, id)
if err != nil && strings.Contains(err.Error(), "plugin lookup error") {
    log.Warn("plugin denormalization failed; serving un-enriched volume", "err", err)
    return vol, nil // degrade gracefully
}

When it happens

Trigger: Reading a CSIVolume (CSIVolumeByID / volume list denormalization) whose PluginID causes CSIPluginByIDTxn's txn.FirstWatch on csi_plugins to return an error — memdb index failure, not a missing row.

Common situations: State store index corruption, version-skewed state database, or bugs in custom forked state store code with altered csi_plugins indexes.

Related errors

• csi_plugin lookup failed
• error parsing: root should be an object
• alloc lookup failed
• cannot specify Accessor ID
• csi_plugin lookup error

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/19b0e94cb7734906. Report an issue: GitHub.

Appendix: source

Thrown at nomad/state/state_store.go:3030

// allocations.
func (s *StateStore) CSIVolumeDenormalizePlugins(ws memdb.WatchSet, vol *structs.CSIVolume) (*structs.CSIVolume, error) {
	if vol == nil {
		return nil, nil
	}
	txn := s.db.ReadTxn()
	defer txn.Abort()
	return s.csiVolumeDenormalizePluginsTxn(txn, vol)
}

// csiVolumeDenormalizePluginsTxn implements
// CSIVolumeDenormalizePlugins, inside a transaction.
func (s *StateStore) csiVolumeDenormalizePluginsTxn(txn Txn, vol *structs.CSIVolume) (*structs.CSIVolume, error) {
	if vol == nil {
		return nil, nil
	}
	plug, err := s.CSIPluginByIDTxn(txn, nil, vol.PluginID)
	if err != nil {
		return nil, fmt.Errorf("plugin lookup error: %s %v", vol.PluginID, err)
	}
	if plug == nil {
		vol.ControllersHealthy = 0
		vol.NodesHealthy = 0
		vol.Schedulable = false
		return vol, nil
	}

	vol.Provider = plug.Provider
	vol.ProviderVersion = plug.Version
	vol.ControllerRequired = plug.ControllerRequired
	vol.ControllersHealthy = plug.ControllersHealthy
	vol.NodesHealthy = plug.NodesHealthy

	// This value may be stale, but stale is ok
	vol.ControllersExpected = plug.ControllersExpected
	vol.NodesExpected = plug.NodesExpected

View on GitHub (pinned to 482b49bf1a)