quickwit-oss/quickwit · error

timed out after {} while waiting for compactor to finish dec

Error message

timed out after {} while waiting for compactor to finish decommissioning

What it means

The decommission wait loop races the compactor's departure against a sleep of `timeout_after`. When the sleep fires first — the compactor never finished decommissioning within the budget — DECOMMISSION_FAILED is incremented, the timeout is logged, and this anyhow error is returned instead of pretending the shutdown succeeded.

Source

Thrown at quickwit/quickwit-compaction/src/compactor_supervisor.rs:485

                    error!(
                        %error,
                        "failed to decommission compactor after {}",
                        timeout_after.pretty_display()
                    );
                    Err(anyhow::anyhow!(
                        "failed to decommission compactor after {}",
                        timeout_after.pretty_display()
                    ))
                }
            }
        }
        _ = &mut sleep => {
            DECOMMISSION_FAILED.inc();
            error!(
                "timed out after {} while waiting for compactor to finish decommissioning",
                timeout_after.pretty_display()
            );
            Err(anyhow::anyhow!(
                "timed out after {} while waiting for compactor to finish decommissioning",
                timeout_after.pretty_display()
            ))
        }
    }
}

#[cfg(test)]
mod tests {
    use std::collections::HashSet;
    use std::num::NonZeroUsize;

    use quickwit_actors::Universe;
    use quickwit_common::temp_dir::TempDirectory;
    use quickwit_proto::compaction::{
        CompactionPlannerServiceClient, MockCompactionPlannerService,
    };
    use quickwit_proto::metastore::{MetastoreServiceClient, MockMetastoreService};

View on GitHub (pinned to a39730c5cd)

Solutions

1. Raise the decommission timeout to cover the longest expected compaction workload.
2. Ensure cluster membership (chitchat) is healthy so departure is actually detected.
3. Drain/finish long compaction tasks before initiating decommission.

Example fix

// before
wait_for_compactor_decommission(&mut compactor, Duration::from_secs(10)).await?;
// after
wait_for_compactor_decommission(&mut compactor, Duration::from_secs(300)).await?;

match wait_for_compactor_decommission(&mut compactor, timeout).await {
    Err(e) if e.to_string().contains("timed out after") => {
        warn!("compactor still draining; extending grace period");
        wait_for_compactor_decommission(&mut compactor, Duration::from_secs(600)).await?;
    }
    other => other?,
}

When it happens

Trigger: Calling wait_for_compactor_decommission while the compactor keeps running work (or its departure is never observed) until the timeout elapses and the `sleep` branch of the select loop wins.

Common situations: Node shutdown with an overloaded compactor still executing merges; network partition preventing cluster membership updates from registering the departure; too-short configured grace period.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• failed to decommission compactor after {}
• lambda function '{}' did not become ready within {} seconds
• timed out while waiting for ingester to transition to status
• internal timeout on get_slice
• Receiver lives longer than sender

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/fcaaa4d02648ab04. Report an issue: GitHub.