{"record":{"id":"fcaaa4d02648ab04","repo":"quickwit-oss/quickwit","slug":"timed-out-after-while-waiting-for-compactor-to","errorCode":null,"errorMessage":"timed out after {} while waiting for compactor to finish decommissioning","messagePattern":"timed out after (.+?) while waiting for compactor to finish decommissioning","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-compaction/src/compactor_supervisor.rs","lineNumber":485,"sourceCode":"                    error!(\n                        %error,\n                        \"failed to decommission compactor after {}\",\n                        timeout_after.pretty_display()\n                    );\n                    Err(anyhow::anyhow!(\n                        \"failed to decommission compactor after {}\",\n                        timeout_after.pretty_display()\n                    ))\n                }\n            }\n        }\n        _ = &mut sleep => {\n            DECOMMISSION_FAILED.inc();\n            error!(\n                \"timed out after {} while waiting for compactor to finish decommissioning\",\n                timeout_after.pretty_display()\n            );\n            Err(anyhow::anyhow!(\n                \"timed out after {} while waiting for compactor to finish decommissioning\",\n                timeout_after.pretty_display()\n            ))\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::HashSet;\n    use std::num::NonZeroUsize;\n\n    use quickwit_actors::Universe;\n    use quickwit_common::temp_dir::TempDirectory;\n    use quickwit_proto::compaction::{\n        CompactionPlannerServiceClient, MockCompactionPlannerService,\n    };\n    use quickwit_proto::metastore::{MetastoreServiceClient, MockMetastoreService};","sourceCodeStart":467,"sourceCodeEnd":503,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-compaction/src/compactor_supervisor.rs#L467-L503","documentation":"The decommission wait loop races the compactor's departure against a sleep of `timeout_after`. When the sleep fires first — the compactor never finished decommissioning within the budget — DECOMMISSION_FAILED is incremented, the timeout is logged, and this anyhow error is returned instead of pretending the shutdown succeeded.","triggerScenarios":"Calling wait_for_compactor_decommission while the compactor keeps running work (or its departure is never observed) until the timeout elapses and the `sleep` branch of the select loop wins.","commonSituations":"Node shutdown with an overloaded compactor still executing merges; network partition preventing cluster membership updates from registering the departure; too-short configured grace period.","solutions":["Raise the decommission timeout to cover the longest expected compaction workload.","Ensure cluster membership (chitchat) is healthy so departure is actually detected.","Drain/finish long compaction tasks before initiating decommission."],"exampleFix":"// before\nwait_for_compactor_decommission(&mut compactor, Duration::from_secs(10)).await?;\n// after\nwait_for_compactor_decommission(&mut compactor, Duration::from_secs(300)).await?;","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match wait_for_compactor_decommission(&mut compactor, timeout).await {\n    Err(e) if e.to_string().contains(\"timed out after\") => {\n        warn!(\"compactor still draining; extending grace period\");\n        wait_for_compactor_decommission(&mut compactor, Duration::from_secs(600)).await?;\n    }\n    other => other?,\n}","preventionTips":["Set generous shutdown grace periods for compactor-heavy nodes.","Ensure chitchat membership is healthy so departures are detected promptly.","Stop admitting new compaction work before decommissioning."],"tags":["compactor","shutdown","timeout"],"backgroundTag":"request-timeout","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}