influxdata/influxdb · critical

token to be updated

Error message

token to be updated

What it means

A panic from `.expect()` in the token-update path of catalog batch replay. After deciding `needs_update` (an existing token must be refreshed), `update_token()` failed to find the token, meaning the token disappeared from `self.tokens` between the existence check and the update. The catalog treats this as an integrity violation and aborts the process.

Solutions

1. Serialize catalog writes so token scans and updates are atomic with respect to each other.
2. Check for a concurrent DeleteToken/rotate path removing the token mid-batch.
3. Re-derive catalog state from a consistent snapshot if the token set is corrupted.
4. Convert the expect into a warn-and-skip (or re-insert) if the race is expected to be benign.

Example fix

// before
self.tokens
    .update_token(token_id, (*updated_token).clone())
    .expect("token to be updated");
// after
if self.tokens.update_token(token_id, (*updated_token).clone()).is_none() {
    warn!(?token_id, "token missing at update time; re-inserting");
    self.tokens.insert_token((*updated_token).clone());
}

// confirm the token still exists before update
if tokens.get(token_id).is_none() { /* re-insert, skip, or error out gracefully */ }

fn token_exists(tokens: &Tokens, id: TokenId) -> bool { tokens.get(id).is_some() }

When it happens

Trigger: Replaying a batch that mutates an existing token where the token_id present in the earlier scan is missing from the tokens repository at update time; concurrent token deletion during replay.

Common situations: Racing token revocation and token-update operations, duplicated/overlapping batch replay, or catalog snapshot/WAL inconsistency hiding the token.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• existing database should be updated
• existing node should update
• there should not already be a node
• {0}
• auto field family exists

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/9aa9f7f1ff5d68d5. Report an issue: GitHub.

Appendix: source

Thrown at influxdb3_catalog/src/catalog/versions/v2.rs:1890

                            // Capture the database name with deletion metadata
                            if let Some(ref mut resource_names) = permission.resource_names {
                                resource_names.entry(db_id.to_string()).or_insert_with(|| {
                                    influxdb3_authz::ResourceMetadata {
                                        name: db_name.to_string(),
                                        deleted: true,
                                    }
                                });
                                needs_update = true;
                            }
                        }
                    }

                    if needs_update {
                        // Update the token in the repository
                        self.tokens
                            .update_token(token_id, (*updated_token).clone())
                            .expect("token to be updated");
                    }
                }
            }
        }

        // Now proceed with the normal database batch processing
        if let Some(db) = self.databases.get_by_id(&database_batch.database_id) {
            let Some(new_db) = DatabaseSchema::new_if_updated_from_batch(&db, database_batch)?
            else {
                return Ok(false);
            };
            self.databases
                .update(db.id, new_db)
                .expect("existing database should be updated");
        } else {
            let new_db = DatabaseSchema::new_from_batch(database_batch)?;
            self.databases
                .insert(new_db.id, new_db)

View on GitHub (pinned to 06200ef96b)