risingwavelabs/risingwave · error · BackupError

too many existent meta snapshots, expect at most {}

Error message

too many existent meta snapshots, expect at most {}

What it means

Before creating a new snapshot, start_backup_job counts existing snapshots in the backup manifest and refuses to proceed if the count exceeds MAX_META_SNAPSHOT_NUM. This limits MetaSnapshotManifest size and backup storage growth.

Source

Thrown at src/meta/src/backup_restore/backup_manager.rs:223

                job.job_id
            ));
        }
        // The reasons to limit number of meta snapshot are:
        // 1. limit size of `MetaSnapshotManifest`, which is kept in memory by
        // `ObjectStoreMetaSnapshotStorage`.
        // 2. limit number of pinned SSTs returned by
        // `list_pinned_ssts`, which subsequently is used by GC.
        const MAX_META_SNAPSHOT_NUM: usize = 100;
        let current_number = self
            .backup_store
            .load()
            .0
            .manifest()
            .await
            .snapshot_metadata
            .len();
        if current_number > MAX_META_SNAPSHOT_NUM {
            bail!(format!(
                "too many existent meta snapshots, expect at most {}",
                MAX_META_SNAPSHOT_NUM
            ))
        }

        let job_id = next_meta_backup_id(&self.env).await?;
        self.latest_job_info
            .store(Arc::new((job_id, BackupJobStatus::Running, "".into())));
        let hummock_version_safe_point = self.hummock_manager.register_safe_point().await;
        // Ideally `BackupWorker` and its r/w IO can be made external to meta node.
        // The justification of keeping `BackupWorker` in meta node are:
        // - It makes meta node the only writer of backup storage, which eases implementation.
        // - It's likely meta store is deployed in the same node with meta node.
        // - IO volume of metadata snapshot is not expected to be large.
        // - Backup job is not expected to be frequent.
        BackupWorker::new(self.clone()).start(job_id, remarks);
        let job_handle = BackupJobHandle::new(job_id, hummock_version_safe_point);
        *guard = Some(job_handle);

View on GitHub (pinned to 6469eb736d)

Solutions

1. Delete/retire old meta snapshots (e.g. `risectl meta delete-backup` for unneeded snapshot ids) then retry
2. Enable/verify automatic backup cleanup so stale snapshots are removed
3. Temporarily raise the retention limit only if storage and manifest size permit

Example fix

// before
risectl meta backup  # fails: too many existent meta snapshots
// after
risectl meta delete-backup <old_id>  # prune stale snapshots
risectl meta backup

let count = backup_store.manifest().await.snapshot_metadata.len();
if count > MAX_META_SNAPSHOT_NUM { /* prune snapshots first */ }

When it happens

Trigger: Calling start_backup_job when the backup store manifest already contains more than MAX_META_SNAPSHOT_NUM snapshot metadata entries.

Common situations: Long-running clusters where old meta snapshots were never pruned; disable-auto-backup-removal configs retaining all history.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• concurrent backup job is not supported: existent job {}
• inconsistent hummock version: expected {}, actual {}
• snapshot id {} not found
• backup job status not found: job {}, {}
• backup job failed: job {}, {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/330d7302a81447bd. Report an issue: GitHub.