What causes "acl token lookup failed: missing secret id" in hashicorp/nomad?

Guard with a non-empty check (after TrimSpace) before the store call.

acl token lookup failed: missing secret id

Error message

acl token lookup failed: missing secret id

What it means

Validation error from StateStore.ACLTokenBySecretID: an empty secret ID was supplied. SecretID is the credential users authenticate with (the 'secret' index on acl_token), so the store rejects empty input before querying. Like 2747, it signals a caller-input problem.

Source

Thrown at nomad/state/state_store.go:6466

	if existing == nil {
		return nil, nil
	}

	// Assert the token type which allows us to perform additional work on the
	// token that is needed before returning the call.
	token := existing.(*structs.ACLToken)

	// Handle potential staleness of ACL role links.
	if token, err = s.fixTokenRoleLinks(txn, token); err != nil {
		return nil, err
	}
	return token, nil
}

// ACLTokenBySecretID is used to lookup a token by secret ID
func (s *StateStore) ACLTokenBySecretID(ws memdb.WatchSet, secretID string) (*structs.ACLToken, error) {
	if secretID == "" {
		return nil, fmt.Errorf("acl token lookup failed: missing secret id")
	}

	txn := s.db.ReadTxn()

	watchCh, existing, err := txn.FirstWatch("acl_token", "secret", secretID)
	if err != nil {
		return nil, fmt.Errorf("acl token lookup failed: %v", err)
	}
	ws.Add(watchCh)

	// If the existing token is nil, this indicates it does not exist in state.
	if existing == nil {
		return nil, nil
	}

	// Assert the token type which allows us to perform additional work on the
	// token that is needed before returning the call.
	token := existing.(*structs.ACLToken)

View on GitHub (pinned to 482b49bf1a)

Solutions

Guard with a non-empty check (after TrimSpace) before the store call.
Return a clear auth error (403/permission denied) at the RPC layer for missing tokens instead of the store sentinel.
Fix client configuration so the token is actually set (NOMAD_TOKEN, agent config, etc.).

Example fix

// before
token, err := store.ACLTokenBySecretID(ws, header.Get("X-Nomad-Token"))
// after
secret := header.Get("X-Nomad-Token")
if secret == "" {
    return structs.NewErrRPCCoded(403, "Permission denied")
}
token, err := store.ACLTokenBySecretID(ws, secret)

Defensive patterns

Strategy: validation

Validate before calling

func canLookupBySecret(secretID string) bool {
    return strings.TrimSpace(secretID) != ""
}
// before calling:
// if !canLookupBySecret(secretID) { return errors.New("missing secret id") }

Type guard

func hasSecretID(t *struct.ACLToken) bool {
    return t != nil && t.SecretID != ""
}

Try / catch

if secretID == "" {
    return structs.NewErrRPCCoded(403, "Permission denied")
}
token, err := store.ACLTokenBySecretID(ws, secretID)
if err != nil { return err }

Prevention

Ensure NOMAD_TOKEN / X-Nomad-Token is actually set in clients and CI
Trim and validate token strings before resolution
Fail closed (permission denied) on empty credentials at the handler layer
Cover the empty-secret path in auth middleware tests

When it happens

Trigger: Calling ACLTokenBySecretID with "" — empty ACL token from a client request, unset environment/config value, or a handler that did not validate before store access.

Common situations: nomad CLI/jobs submitted with NOMAD_TOKEN=""; API clients omitting X-Nomad-Token; plugins resolving tokens from empty fields; tests exercising the empty path.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5016f82ea091f119. Report an issue: GitHub.