quickwit-oss/quickwit · error

failed to decommission compactor after {}

Error message

failed to decommission compactor after {}

What it means

`wait_for_compactor_decommission` waits for a compactor actor to drain and be removed from the cluster after a decommission request. If the decommission flow completes but reports an error after the timeout budget elapsed, it increments DECOMMISSION_FAILED, logs with the elapsed pretty duration, and returns this anyhow error to the caller (e.g. shutdown supervision).

Source

Thrown at quickwit/quickwit-compaction/src/compactor_supervisor.rs:472

    tokio::select! {
        result = status_rx.wait_for(|status| *status == CompactorStatus::Decommissioned) => {
            match result {
                Ok(_) => {
                    DECOMMISSION_SUCCEEDED.inc();
                    info!(
                        "compactor decommissioned successfully in {}",
                        now.elapsed().pretty_display()
                    );
                    Ok(())
                }
                Err(error) => {
                    DECOMMISSION_FAILED.inc();
                    error!(
                        %error,
                        "failed to decommission compactor after {}",
                        timeout_after.pretty_display()
                    );
                    Err(anyhow::anyhow!(
                        "failed to decommission compactor after {}",
                        timeout_after.pretty_display()
                    ))
                }
            }
        }
        _ = &mut sleep => {
            DECOMMISSION_FAILED.inc();
            error!(
                "timed out after {} while waiting for compactor to finish decommissioning",
                timeout_after.pretty_display()
            );
            Err(anyhow::anyhow!(
                "timed out after {} while waiting for compactor to finish decommissioning",
                timeout_after.pretty_display()
            ))
        }
    }

View on GitHub (pinned to a39730c5cd)

Solutions

1. Increase the decommission/shutdown timeout so long compaction tasks can finish draining.
2. Check logs from the compactor actor for the underlying error after decommission.
3. Retry the decommission; if persistent, inspect control-plane indexing task placement.

match wait_for_compactor_decommission(&mut compactor, timeout).await {
    Err(e) if e.to_string().starts_with("failed to decommission compactor after") => {
        warn!("decommission failed, retrying with longer timeout");
        wait_for_compactor_decommission(&mut compactor, timeout * 2).await?;
    }
    other => other?,
}

When it happens

Trigger: Calling wait_for_compactor_decommission (or shutting down the compactor supervisor) when the underlying decommission call returns an error after `timeout_after` has elapsed — the compactor failed to gracefully stop/drain in time.

Common situations: Compactor busy with long-running merge/compact operations during node shutdown; slow cluster membership updates delaying the departure detection; faults in the control plane preventing task evacuation.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• timed out after {} while waiting for compactor to finish dec
• lambda function '{}' did not become ready within {} seconds
• timed out while waiting for ingester to transition to status
• internal timeout on get_slice
• Receiver lives longer than sender

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/4af940dcb7048ca7. Report an issue: GitHub.