{"record":{"id":"4af940dcb7048ca7","repo":"quickwit-oss/quickwit","slug":"failed-to-decommission-compactor-after","errorCode":null,"errorMessage":"failed to decommission compactor after {}","messagePattern":"failed to decommission compactor after (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-compaction/src/compactor_supervisor.rs","lineNumber":472,"sourceCode":"    tokio::select! {\n        result = status_rx.wait_for(|status| *status == CompactorStatus::Decommissioned) => {\n            match result {\n                Ok(_) => {\n                    DECOMMISSION_SUCCEEDED.inc();\n                    info!(\n                        \"compactor decommissioned successfully in {}\",\n                        now.elapsed().pretty_display()\n                    );\n                    Ok(())\n                }\n                Err(error) => {\n                    DECOMMISSION_FAILED.inc();\n                    error!(\n                        %error,\n                        \"failed to decommission compactor after {}\",\n                        timeout_after.pretty_display()\n                    );\n                    Err(anyhow::anyhow!(\n                        \"failed to decommission compactor after {}\",\n                        timeout_after.pretty_display()\n                    ))\n                }\n            }\n        }\n        _ = &mut sleep => {\n            DECOMMISSION_FAILED.inc();\n            error!(\n                \"timed out after {} while waiting for compactor to finish decommissioning\",\n                timeout_after.pretty_display()\n            );\n            Err(anyhow::anyhow!(\n                \"timed out after {} while waiting for compactor to finish decommissioning\",\n                timeout_after.pretty_display()\n            ))\n        }\n    }","sourceCodeStart":454,"sourceCodeEnd":490,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-compaction/src/compactor_supervisor.rs#L454-L490","documentation":"`wait_for_compactor_decommission` waits for a compactor actor to drain and be removed from the cluster after a decommission request. If the decommission flow completes but reports an error after the timeout budget elapsed, it increments DECOMMISSION_FAILED, logs with the elapsed pretty duration, and returns this anyhow error to the caller (e.g. shutdown supervision).","triggerScenarios":"Calling wait_for_compactor_decommission (or shutting down the compactor supervisor) when the underlying decommission call returns an error after `timeout_after` has elapsed — the compactor failed to gracefully stop/drain in time.","commonSituations":"Compactor busy with long-running merge/compact operations during node shutdown; slow cluster membership updates delaying the departure detection; faults in the control plane preventing task evacuation.","solutions":["Increase the decommission/shutdown timeout so long compaction tasks can finish draining.","Check logs from the compactor actor for the underlying error after decommission.","Retry the decommission; if persistent, inspect control-plane indexing task placement."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match wait_for_compactor_decommission(&mut compactor, timeout).await {\n    Err(e) if e.to_string().starts_with(\"failed to decommission compactor after\") => {\n        warn!(\"decommission failed, retrying with longer timeout\");\n        wait_for_compactor_decommission(&mut compactor, timeout * 2).await?;\n    }\n    other => other?,\n}","preventionTips":["Size the decommission timeout to the longest-running compaction task.","Monitor DECOMMISSION_FAILED metric for recurring drain failures."],"tags":["compactor","shutdown","timeout"],"backgroundTag":"request-timeout","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}