risingwavelabs/risingwave · warning

failed to start GC due to an ongoing process

Error message

failed to start GC due to an ongoing process

What it means

Full GC (manual full garbage collection of Hummock objects) is guarded by a try_start/stop state machine so only one full-GC process runs at a time. start_full_gc returns this error when full_gc_state.try_start() fails, i.e. another full GC is already in progress. The scopeguard ensures the state is reset when the running GC finishes, so the error is transient.

Source

Thrown at src/meta/src/hummock/manager/gc.rs:255

        let tracked_object_ids: HashSet<HummockObjectId> =
            versioning.get_tracked_object_ids(min_pinned_version_id);
        let to_delete = object_ids
            .filter(|object_id| !tracked_object_ids.contains(object_id))
            .collect_vec();
        self.write_gc_history(to_delete.iter().copied()).await?;
        Ok(to_delete)
    }

    /// LIST object store and DELETE stale objects, in batches.
    /// GC can be very slow. Spawn a dedicated tokio task for it.
    pub async fn start_full_gc(
        &self,
        object_retention_time: Duration,
        prefix: Option<String>,
        backup_manager: Option<BackupManagerRef>,
    ) -> Result<()> {
        if !self.full_gc_state.try_start() {
            return Err(anyhow::anyhow!("failed to start GC due to an ongoing process").into());
        }
        let _guard = scopeguard::guard(self.full_gc_state.clone(), |full_gc_state| {
            full_gc_state.stop()
        });
        self.metrics.full_gc_trigger_count.inc();
        let object_retention_time = cmp::max(
            object_retention_time,
            Duration::from_secs(self.env.opts.min_sst_retention_time_sec),
        );
        let limit = self.env.opts.full_gc_object_limit;
        let mut start_after = None;
        let object_retention_watermark = self
            .now()
            .await?
            .saturating_sub(object_retention_time.as_secs());
        let mut total_object_count = 0;
        let mut total_object_size = 0;
        tracing::info!(

View on GitHub (pinned to 6469eb736d)

Solutions

1. Wait for the ongoing full GC to finish and check its progress via full GC state/metrics before retrying.
2. If the running GC is stuck, inspect meta logs and restart the meta node to reset full_gc_state, then retry.
3. Add retry-with-backoff or a check of the GC state before issuing another full_gc request in automation.

// treat as busy; retry with backoff
match meta.start_full_gc(retention, prefix, backup).await {
    Err(e) if e.to_string().contains("ongoing process") => {
        tokio::time::sleep(Duration::from_secs(60)).await; // then retry
    }
    other => other,
}

When it happens

Trigger: Calling start_full_gc (e.g. via the hummock admin RPC or risingwave ctl full-gc) while a previous full GC is still running. Repeatedly invoking full GC without waiting for the prior run to complete.

Common situations: An operator re-triggers full GC because they think the first attempt failed; an automated script retrying full GC on a short interval; a very slow full GC on a huge object store overlapping the next scheduled trigger.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• too many inflight time travel queries, max_inflight_time_tra
• object {object_id} is rejected from being committed since it
• version delta {} not found
• prev_version {} not found
• Wait epoch error: {0}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/74da4f764d059b75. Report an issue: GitHub.