hashicorp/nomad · error

token lookup failed: %v

Error message

token lookup failed: %v

What it means

This error occurs inside StateStore.UpsertACLTokens when txn.First on the 'acl_token' table by accessor ID fails. It means the memdb read of the existing token errored (index/table problem), aborting the whole token upsert transaction. The upsert transaction is rolled back and the Raft apply fails.

Source

Thrown at nomad/state/state_store.go:6385

	return iter, nil
}

// UpsertACLTokens is used to create or update a set of ACL tokens
func (s *StateStore) UpsertACLTokens(msgType structs.MessageType, index uint64, tokens []*structs.ACLToken) error {
	txn := s.db.WriteTxnMsgT(msgType, index)
	defer txn.Abort()

	for _, token := range tokens {
		// Ensure the policy hash is non-nil. This should be done outside the state store
		// for performance reasons, but we check here for defense in depth.
		if len(token.Hash) == 0 {
			token.SetHash()
		}

		// Check if the token already exists
		existing, err := txn.First("acl_token", "id", token.AccessorID)
		if err != nil {
			return fmt.Errorf("token lookup failed: %v", err)
		}

		// Update all the indexes
		if existing != nil {
			existTK := existing.(*structs.ACLToken)
			token.CreateIndex = existTK.CreateIndex
			token.ModifyIndex = index

			// Do not allow SecretID or create time to change
			token.SecretID = existTK.SecretID
			token.CreateTime = existTK.CreateTime

		} else {
			token.CreateIndex = index
			token.ModifyIndex = index
		}

		// Update the token

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the Nomad server to reinitialize the in-memory state store from BoltDB.
2. Verify the acl_token table schema (index 'id') matches the running binary; align versions.
3. Capture the wrapped %v cause in logs; if it indicates corruption, restore from the latest state snapshot backup.

Example fix

// before
existing, err := txn.First("acl_token", "id", token.AccessorID)
if err != nil {
    return fmt.Errorf("token lookup failed: %v", err)
}
// after
existing, err := txn.First("acl_token", "id", token.AccessorID)
if err != nil {
    return fmt.Errorf("token lookup failed: %w", err)
}

func validTokenForUpsert(t *structs.ACLToken) bool {
    return t != nil && t.AccessorID != "" && t.SecretID != ""
}

if err := store.UpsertACLTokens(msgType, index, tokens); err != nil {
    if strings.Contains(err.Error(), "token lookup failed") {
        return fmt.Errorf("state store rejected token upsert (schema/store issue): %w", err)
    }
    return err
}

When it happens

Trigger: UpsertACLTokens (e.g., an ACLTokenUpsertRequest via the state store) hitting a memdb error on 'acl_token'/'id' — schema mismatch, nil txn, or corrupted in-memory table; concurrent store shutdown mid-write.

Common situations: State store schema drift after an upgrade; a forked Nomad with modified acl_token indexes; tests using a closed or mis-initialized StateStore.

Related errors

• index update failed: %v
• alloc %q lookup failed: %v
• policy lookup failed: %v
• upserting policy failed: %v
• upserting token failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/759e1aea789e0f9a. Report an issue: GitHub.