What causes "deleting acl token failed: %v" in hashicorp/nomad?

Restart the Nomad server to rebuild the state store from BoltDB.

deleting acl token failed: %v

Error message

deleting acl token failed: %v

What it means

Thrown in StateStore.DeleteACLTokens when txn.DeleteAll on the 'acl_token' table by accessor ID fails. The delete transaction aborts, so no tokens are removed and the Raft log entry fails, surfacing as a failed ACLTokenDelete RPC.

Source

Thrown at nomad/state/state_store.go:6424

		}
	}

	// Update the indexes table
	if err := txn.Insert("index", &IndexEntry{"acl_token", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}
	return txn.Commit()
}

// DeleteACLTokens deletes the tokens with the given accessor ids
func (s *StateStore) DeleteACLTokens(msgType structs.MessageType, index uint64, ids []string) error {
	txn := s.db.WriteTxnMsgT(msgType, index)
	defer txn.Abort()

	// Delete the tokens
	for _, id := range ids {
		if _, err := txn.DeleteAll("acl_token", "id", id); err != nil {
			return fmt.Errorf("deleting acl token failed: %v", err)
		}
	}
	if err := txn.Insert("index", &IndexEntry{"acl_token", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}
	return txn.Commit()
}

// ACLTokenByAccessorID is used to lookup a token by accessor ID
func (s *StateStore) ACLTokenByAccessorID(ws memdb.WatchSet, id string) (*structs.ACLToken, error) {
	if id == "" {
		return nil, fmt.Errorf("acl token lookup failed: missing accessor id")
	}

	txn := s.db.ReadTxn()

	watchCh, existing, err := txn.FirstWatch("acl_token", "id", id)
	if err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

Restart the Nomad server to rebuild the state store from BoltDB.
Confirm the acl_token table defines the 'id' index in the deployed schema version.
Check the wrapped %v cause in logs; restore server state from backup if corruption is indicated.

Defensive patterns

Strategy: try-catch

Validate before calling

func validDeleteIDs(ids []string) bool {
    for _, id := range ids {
        if id == "" { return false }
    }
    return len(ids) > 0
}

Try / catch

if err := store.DeleteACLTokens(msgType, index, ids); err != nil {
    if strings.Contains(err.Error(), "deleting acl token failed") {
        return fmt.Errorf("state store token delete aborted: %w", err)
    }
    return err
}

Prevention

Validate IDs are non-empty before the delete RPC
Restart the server on state store errors; do not retry the same Raft write blindly
Keep acl_token 'id' index definitions in sync with the binary
Track delete failures via ACLTokenDelete RPC error metrics

When it happens

Trigger: DeleteACLTokens with a memdb error on 'acl_token'/'id' — schema drift for the table, store closed concurrently, or internal corruption of the in-memory table.

Common situations: Deleting expired/revoked tokens during a version upgrade with schema skew; forked builds with altered acl_token indexes; tests using a torn-down state store.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/deb391c641e7f89f. Report an issue: GitHub.