What causes "acl policy lookup failed: %v" in hashicorp/nomad?

acl policy lookup failed: %v

Error message

acl policy lookup failed: %v

What it means

Thrown by StateStore.ACLPolicyByName when the WatchFirst/FirstWatch lookup of an ACL policy by exact name fails in memdb. This is a read-path error: it wraps an internal query failure, not a missing policy (missing policies return nil, nil). Callers (ACL Get Policy RPC, token resolution) surface it as a 500-style internal error.

Source

Thrown at nomad/state/state_store.go:6308

	// Delete the policy
	for _, name := range names {
		if _, err := txn.DeleteAll("acl_policy", "id", name); err != nil {
			return fmt.Errorf("deleting acl policy failed: %v", err)
		}
	}
	if err := txn.Insert("index", &IndexEntry{"acl_policy", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}
	return txn.Commit()
}

// ACLPolicyByName is used to lookup a policy by name
func (s *StateStore) ACLPolicyByName(ws memdb.WatchSet, name string) (*structs.ACLPolicy, error) {
	txn := s.db.ReadTxn()

	watchCh, existing, err := txn.FirstWatch("acl_policy", "id", name)
	if err != nil {
		return nil, fmt.Errorf("acl policy lookup failed: %v", err)
	}
	ws.Add(watchCh)

	if existing != nil {
		return existing.(*structs.ACLPolicy), nil
	}
	return nil, nil
}

// ACLPolicyByNamePrefix is used to lookup policies by prefix
func (s *StateStore) ACLPolicyByNamePrefix(ws memdb.WatchSet, prefix string) (memdb.ResultIterator, error) {
	txn := s.db.ReadTxn()

	iter, err := txn.Get("acl_policy", "id_prefix", prefix)
	if err != nil {
		return nil, fmt.Errorf("acl policy lookup failed: %v", err)
	}
	ws.Add(iter.WatchCh())

View on GitHub (pinned to 482b49bf1a)

Solutions

Retry the read request
Confirm the exact policy name with nomad acl policy list (though name mismatch yields 'not found', not this error)
Check server health/logs for memdb errors; restart the server if needed
Restore from snapshot if state corruption is indicated

Defensive patterns

Strategy: try-catch

Validate before calling

// Distinguish 'not found' (safe) from internal failure using the read API
const policy = await nomad.get(`/v1/acl/policy/${encodeURIComponent(name)}`).catch(e => {
  if (e.status === 404) return null; // policy absent is normal
  throw e;                           // 500-class: internal lookup failure
});

Type guard

function isNotFound(e) {
  return e && (e.status === 404 || /not found/i.test(e.message || ''));
}

Try / catch

try {
  const policy = await nomad.get(`/v1/acl/policy/${encodeURIComponent(name)}`);
} catch (e) {
  if (isNotFound(e)) return null;
  if (String(e).includes('acl policy lookup failed')) return retryWithBackoff(readPolicy, name);
  throw e;
}

Prevention

Treat 404 and 5xx differently: 404 is expected for absent policies
Retry reads with backoff on 500-class failures
Watch server memory on clusters with heavy blocking-query ACL traffic
Restore from snapshot if lookup failures persist across restarts

When it happens

Trigger: ACL policy read by name (GET /v1/acl/policy/:name, nomad acl policy info) while the memdb FirstWatch query on acl_policy errors — internal store failure, corruption, or memory exhaustion; also hit indirectly when resolving policies attached to ACL tokens.

Common situations: Reading policies on a degraded/corrupted server; heavy blocking-query load exhausting memory; issues after partial snapshot restore.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6ddcc89adbe8204c. Report an issue: GitHub.