hashicorp/nomad · error

upserting token failed: %v

Error message

upserting token failed: %v

What it means

Thrown when txn.Insert of an ACLToken into the 'acl_token' table fails during UpsertACLTokens. memdb Insert fails if the object is not of the registered table type or violates an index (e.g., missing required index fields), causing the whole transaction to abort.

Source

Thrown at nomad/state/state_store.go:6405

		// Update all the indexes
		if existing != nil {
			existTK := existing.(*structs.ACLToken)
			token.CreateIndex = existTK.CreateIndex
			token.ModifyIndex = index

			// Do not allow SecretID or create time to change
			token.SecretID = existTK.SecretID
			token.CreateTime = existTK.CreateTime

		} else {
			token.CreateIndex = index
			token.ModifyIndex = index
		}

		// Update the token
		if err := txn.Insert("acl_token", token); err != nil {
			return fmt.Errorf("upserting token failed: %v", err)
		}
	}

	// Update the indexes table
	if err := txn.Insert("index", &IndexEntry{"acl_token", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}
	return txn.Commit()
}

// DeleteACLTokens deletes the tokens with the given accessor ids
func (s *StateStore) DeleteACLTokens(msgType structs.MessageType, index uint64, ids []string) error {
	txn := s.db.WriteTxnMsgT(msgType, index)
	defer txn.Abort()

	// Delete the tokens
	for _, id := range ids {
		if _, err := txn.DeleteAll("acl_token", "id", id); err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure the ACLToken has non-empty AccessorID and SecretID and call token.SetHash() before upsert (as the store does when CreateTime is unset).
2. Construct tokens via the ACL token bootstrap/create API rather than manual structs.
3. Check that the binary and state schema versions match; inspect logs for the wrapped %v cause.

Example fix

// before
token := &structs.ACLToken{SecretID: secret}
s.UpsertACLTokens(msgType, index, []*structs.ACLToken{token})
// after
token := &structs.ACLToken{AccessorID: accessor, SecretID: secret}
token.SetHash()
s.UpsertACLTokens(msgType, index, []*structs.ACLToken{token})

func validTokenForUpsert(t *structs.ACLToken) bool {
    return t != nil && t.AccessorID != "" && t.SecretID != ""
}
// filter tokens before calling:
// ok := filterSlice(tokens, validTokenForUpsert)

if err := store.UpsertACLTokens(msgType, index, tokens); err != nil {
    if strings.Contains(err.Error(), "upserting token failed") {
        return fmt.Errorf("invalid ACLToken for acl_token table: %w", err)
    }
    return err
}

When it happens

Trigger: Inserting an *structs.ACLToken whose AccessorID or SecretID is empty/nil, breaking unique-index expectations; passing an object of the wrong type; schema registering different index fields than the token provides.

Common situations: Upserting tokens created manually (not via ACL API) with missing AccessorID/SecretID; a version change that added new required index fields; test harnesses constructing ACLToken literals.

Related errors

• token lookup failed: %v
• deleting acl token failed: %v
• ACL policy not found
• ACL role not found
• detected corrupted token within the state store: missing rol

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/63b0ed927a3cdd07. Report an issue: GitHub.