{"record":{"id":"74da4f764d059b75","repo":"risingwavelabs/risingwave","slug":"failed-to-start-gc-due-to-an-ongoing-process","errorCode":null,"errorMessage":"failed to start GC due to an ongoing process","messagePattern":"failed to start GC due to an ongoing process","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/meta/src/hummock/manager/gc.rs","lineNumber":255,"sourceCode":"        let tracked_object_ids: HashSet<HummockObjectId> =\n            versioning.get_tracked_object_ids(min_pinned_version_id);\n        let to_delete = object_ids\n            .filter(|object_id| !tracked_object_ids.contains(object_id))\n            .collect_vec();\n        self.write_gc_history(to_delete.iter().copied()).await?;\n        Ok(to_delete)\n    }\n\n    /// LIST object store and DELETE stale objects, in batches.\n    /// GC can be very slow. Spawn a dedicated tokio task for it.\n    pub async fn start_full_gc(\n        &self,\n        object_retention_time: Duration,\n        prefix: Option<String>,\n        backup_manager: Option<BackupManagerRef>,\n    ) -> Result<()> {\n        if !self.full_gc_state.try_start() {\n            return Err(anyhow::anyhow!(\"failed to start GC due to an ongoing process\").into());\n        }\n        let _guard = scopeguard::guard(self.full_gc_state.clone(), |full_gc_state| {\n            full_gc_state.stop()\n        });\n        self.metrics.full_gc_trigger_count.inc();\n        let object_retention_time = cmp::max(\n            object_retention_time,\n            Duration::from_secs(self.env.opts.min_sst_retention_time_sec),\n        );\n        let limit = self.env.opts.full_gc_object_limit;\n        let mut start_after = None;\n        let object_retention_watermark = self\n            .now()\n            .await?\n            .saturating_sub(object_retention_time.as_secs());\n        let mut total_object_count = 0;\n        let mut total_object_size = 0;\n        tracing::info!(","sourceCodeStart":237,"sourceCodeEnd":273,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/hummock/manager/gc.rs#L237-L273","documentation":"Full GC (manual full garbage collection of Hummock objects) is guarded by a try_start/stop state machine so only one full-GC process runs at a time. start_full_gc returns this error when full_gc_state.try_start() fails, i.e. another full GC is already in progress. The scopeguard ensures the state is reset when the running GC finishes, so the error is transient.","triggerScenarios":"Calling start_full_gc (e.g. via the hummock admin RPC or risingwave ctl full-gc) while a previous full GC is still running. Repeatedly invoking full GC without waiting for the prior run to complete.","commonSituations":"An operator re-triggers full GC because they think the first attempt failed; an automated script retrying full GC on a short interval; a very slow full GC on a huge object store overlapping the next scheduled trigger.","solutions":["Wait for the ongoing full GC to finish and check its progress via full GC state/metrics before retrying.","If the running GC is stuck, inspect meta logs and restart the meta node to reset full_gc_state, then retry.","Add retry-with-backoff or a check of the GC state before issuing another full_gc request in automation."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// treat as busy; retry with backoff\nmatch meta.start_full_gc(retention, prefix, backup).await {\n    Err(e) if e.to_string().contains(\"ongoing process\") => {\n        tokio::time::sleep(Duration::from_secs(60)).await; // then retry\n    }\n    other => other,\n}","preventionTips":["Check full-GC state/metrics before triggering a new full GC.","Schedule full GCs far enough apart to let long runs finish.","Use exponential backoff in automated full-GC scripts.","Monitor full GC duration so intervals can be sized correctly."],"tags":["hummock","full-gc","concurrency","risingwave-meta"],"backgroundTag":"invalid-state-transition","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}