influxdata/influxdb · critical

v2 catalog should exist since checkpoint was found

Error message

v2 catalog should exist since checkpoint was found

What it means

A `.expect(...)` panic inside `check_and_migrate_v2_to_v3`: after confirming a v2 checkpoint exists, the code loads the v2 catalog and asserts it must deserialize successfully. It fires when the persisted v2 catalog bytes are missing/corrupt despite a checkpoint marker being present, i.e. a broken invariant between the checkpoint and the catalog object.

Solutions

1. Inspect the object store at the v2 prefix and verify the catalog object referenced by the checkpoint exists and is complete.
2. Restore the missing/corrupt v2 catalog object from backup or re-copy it from a healthy node.
3. If v2 data is not needed, remove the v2 checkpoint so the migration returns NothingToMigrate instead of panicking.
4. Re-run migration from a consistent snapshot of the store; report persistent corruption to InfluxData since this is an unrecoverable panic path.

// This is a panic, not a returned error — run migration in a worker and catch_unwind
let result = std::panic::catch_unwind(|| check_and_migrate_v2_to_v3(prefix, store));
if result.is_err() { restore_v2_catalog_from_backup_or_abort(); }

When it happens

Trigger: Running the v2->v3 catalog migration when the object store has a v2 checkpoint but `load_catalog` returns None or fails to deserialize — e.g. truncated/partially written catalog object, deleted/overwritten catalog object, or mismatched store prefix.

Common situations: Interrupted writes to object storage (network drop mid-PUT), manual tampering/cleanup of catalog objects, restoring a backup that contains the checkpoint but not the catalog, or pointing a new node at a bucket with stale/partial v2 data.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• generation duration overflows u64 nanoseconds
• row_delete_predicate_version exceeds u64
• column id in series key should be valid
• duration not to overflow
• last cache size overflows u64

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/2483ff4c92843c66. Report an issue: GitHub.

Appendix: source

Thrown at influxdb3_catalog/src/catalog/migrations/v3.rs:101

    if v3_store.load_snapshot().await?.is_some() {
        return Ok(MigrationResult::AlreadyMigrated);
    }

    let v2_store = ostore::v2::ObjectStoreCatalog::new(
        Arc::clone(&prefix),
        u64::MAX,
        Arc::clone(&store),
        StorageMode::default(),
    );

    if !v2_store.checkpoint_exists().await? {
        return Ok(MigrationResult::NothingToMigrate);
    }

    let v2_catalog = ser::v2::load_catalog(Arc::clone(&prefix), &v2_store)
        .await?
        .expect("v2 catalog should exist since checkpoint was found");

    let catalog_uuid = v2_catalog.catalog_uuid;
    let catalog_sequence = v2_catalog.sequence_number();
    let v2_snapshot = v2_catalog.snapshot();
    let batch = synthesize_records(&v2_snapshot);

    let mut v3_inner = V3InnerCatalog::new(Arc::clone(&prefix), catalog_uuid);
    apply_records(
        batch.as_slice(),
        &mut v3_inner,
        catalog_sequence,
        RestorePreload::empty(),
    )
    .map_err(|e| MigrationError::Unexpected(anyhow::anyhow!(e)))?;
    let v3_snapshot_bytes = v3_inner.create_snapshot();

    if !v2_catalog.has_upgraded {
        let next_sequence = catalog_sequence.next();

View on GitHub (pinned to 06200ef96b)