{"record":{"id":"330d7302a81447bd","repo":"risingwavelabs/risingwave","slug":"too-many-existent-meta-snapshots-expect-at-most","errorCode":null,"errorMessage":"too many existent meta snapshots, expect at most {}","messagePattern":"too many existent meta snapshots, expect at most (.+?)","errorType":"exception","errorClass":"BackupError","httpStatus":null,"severity":"error","filePath":"src/meta/src/backup_restore/backup_manager.rs","lineNumber":223,"sourceCode":"                job.job_id\n            ));\n        }\n        // The reasons to limit number of meta snapshot are:\n        // 1. limit size of `MetaSnapshotManifest`, which is kept in memory by\n        // `ObjectStoreMetaSnapshotStorage`.\n        // 2. limit number of pinned SSTs returned by\n        // `list_pinned_ssts`, which subsequently is used by GC.\n        const MAX_META_SNAPSHOT_NUM: usize = 100;\n        let current_number = self\n            .backup_store\n            .load()\n            .0\n            .manifest()\n            .await\n            .snapshot_metadata\n            .len();\n        if current_number > MAX_META_SNAPSHOT_NUM {\n            bail!(format!(\n                \"too many existent meta snapshots, expect at most {}\",\n                MAX_META_SNAPSHOT_NUM\n            ))\n        }\n\n        let job_id = next_meta_backup_id(&self.env).await?;\n        self.latest_job_info\n            .store(Arc::new((job_id, BackupJobStatus::Running, \"\".into())));\n        let hummock_version_safe_point = self.hummock_manager.register_safe_point().await;\n        // Ideally `BackupWorker` and its r/w IO can be made external to meta node.\n        // The justification of keeping `BackupWorker` in meta node are:\n        // - It makes meta node the only writer of backup storage, which eases implementation.\n        // - It's likely meta store is deployed in the same node with meta node.\n        // - IO volume of metadata snapshot is not expected to be large.\n        // - Backup job is not expected to be frequent.\n        BackupWorker::new(self.clone()).start(job_id, remarks);\n        let job_handle = BackupJobHandle::new(job_id, hummock_version_safe_point);\n        *guard = Some(job_handle);","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/backup_restore/backup_manager.rs#L205-L241","documentation":"Before creating a new snapshot, start_backup_job counts existing snapshots in the backup manifest and refuses to proceed if the count exceeds MAX_META_SNAPSHOT_NUM. This limits MetaSnapshotManifest size and backup storage growth.","triggerScenarios":"Calling start_backup_job when the backup store manifest already contains more than MAX_META_SNAPSHOT_NUM snapshot metadata entries.","commonSituations":"Long-running clusters where old meta snapshots were never pruned; disable-auto-backup-removal configs retaining all history.","solutions":["Delete/retire old meta snapshots (e.g. `risectl meta delete-backup` for unneeded snapshot ids) then retry","Enable/verify automatic backup cleanup so stale snapshots are removed","Temporarily raise the retention limit only if storage and manifest size permit"],"exampleFix":"// before\nrisectl meta backup  # fails: too many existent meta snapshots\n// after\nrisectl meta delete-backup <old_id>  # prune stale snapshots\nrisectl meta backup","handlingStrategy":"validation","validationCode":"let count = backup_store.manifest().await.snapshot_metadata.len();\nif count > MAX_META_SNAPSHOT_NUM { /* prune snapshots first */ }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Regularly prune old meta snapshots","Enable automatic backup removal policies","Alert when snapshot count approaches MAX_META_SNAPSHOT_NUM"],"tags":["rust","meta","backup","limit-exceeded"],"backgroundTag":"value-out-of-range","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}