hashicorp/nomad · error

plugin lookup error: %s %v

Error message

plugin lookup error: %s %v

What it means

csiVolumeDenormalizePluginsTxn enriches a CSIVolume with health/schedulability data from its plugin. When CSIPluginByIDTxn returns a non-nil error (a memdb lookup failure on the csi_plugins table, not merely a missing plugin), this wrapper propagates it prefixed with the volume's PluginID. Note a nil plugin with nil error is handled gracefully (health zeroed), so this error only fires on genuine lookup failures.

Source

Thrown at nomad/state/state_store.go:3030

// allocations.
func (s *StateStore) CSIVolumeDenormalizePlugins(ws memdb.WatchSet, vol *structs.CSIVolume) (*structs.CSIVolume, error) {
	if vol == nil {
		return nil, nil
	}
	txn := s.db.ReadTxn()
	defer txn.Abort()
	return s.csiVolumeDenormalizePluginsTxn(txn, vol)
}

// csiVolumeDenormalizePluginsTxn implements
// CSIVolumeDenormalizePlugins, inside a transaction.
func (s *StateStore) csiVolumeDenormalizePluginsTxn(txn Txn, vol *structs.CSIVolume) (*structs.CSIVolume, error) {
	if vol == nil {
		return nil, nil
	}
	plug, err := s.CSIPluginByIDTxn(txn, nil, vol.PluginID)
	if err != nil {
		return nil, fmt.Errorf("plugin lookup error: %s %v", vol.PluginID, err)
	}
	if plug == nil {
		vol.ControllersHealthy = 0
		vol.NodesHealthy = 0
		vol.Schedulable = false
		return vol, nil
	}

	vol.Provider = plug.Provider
	vol.ProviderVersion = plug.Version
	vol.ControllerRequired = plug.ControllerRequired
	vol.ControllersHealthy = plug.ControllersHealthy
	vol.NodesHealthy = plug.NodesHealthy

	// This value may be stale, but stale is ok
	vol.ControllersExpected = plug.ControllersExpected
	vol.NodesExpected = plug.NodesExpected

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the server agent to rebuild memdb state from Raft.
2. Check the wrapped inner error (%v) for the actual memdb failure and address it.
3. Ensure the plugin row exists via nomad plugin status / CSIPluginByID; re-register the plugin if needed.
4. Align Nomad versions across the cluster.

// before denormalizing, confirm plugin exists:
plug, err := store.CSIPluginByID(ws, ns, vol.PluginID)
if err != nil { return nil, err }
if plug == nil { vol.Schedulable = false } // missing plugin is not an error

func pluginUsable(p *structs.CSIPlugin) bool {
    return p != nil && p.ControllersHealthy > 0
}

vol, err := store.CSIVolumeByID(ws, ns, id)
if err != nil && strings.Contains(err.Error(), "plugin lookup error") {
    log.Warn("plugin denormalization failed; serving un-enriched volume", "err", err)
    return vol, nil // degrade gracefully
}

When it happens

Trigger: Reading a CSIVolume (CSIVolumeByID / volume list denormalization) whose PluginID causes CSIPluginByIDTxn's txn.FirstWatch on csi_plugins to return an error — memdb index failure, not a missing row.

Common situations: State store index corruption, version-skewed state database, or bugs in custom forked state store code with altered csi_plugins indexes.

Related errors

• error parsing: root should be an object
• csi_plugin lookup failed: %s %v
• cannot specify Accessor ID
• failed to read dynamic plugin registry state: %v
• error getting plugin: %s, %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/19b0e94cb7734906. Report an issue: GitHub.