Hmbown/CodeWhale · warning · anyhow::Error

heavy command canceled while queued for resource admission

Error message

heavy command canceled while queued for resource admission

What it means

acquire_heavy_command_permit_at implements memory-aware admission for heavy commands: it loops trying to lock a slot file (heavy-N.lock) under root, where the effective slot count shrinks with host memory pressure (critical pressure yields zero slots). Each iteration checks the supplied CancellationToken; if the token fires while the command is still queued (no free slot at the current effective limit), this error is returned instead of a permit.

Source

Thrown at crates/tui/src/tools/resource_admission.rs:188

    let probe = HostMemoryProbe;
    acquire_heavy_command_permit_at(&root, limit, cancel, &probe)
        .await
        .map(Some)
}

async fn acquire_heavy_command_permit_at(
    root: &Path,
    limit: usize,
    cancel: Option<&CancellationToken>,
    probe: &dyn MemoryProbe,
) -> Result<HeavyCommandPermit> {
    std::fs::create_dir_all(root)
        .with_context(|| format!("creating resource admission directory {}", root.display()))?;
    let started = Instant::now();

    loop {
        if cancel.is_some_and(|token| token.is_cancelled()) {
            return Err(anyhow!(
                "heavy command canceled while queued for resource admission"
            ));
        }
        // Re-measure each iteration: under memory pressure the effective limit
        // tightens so a saturated host stops admitting new heavy link graphs
        // (#4864 req 7). Critical pressure yields zero slots, so the command
        // waits for recovery instead of snowballing.
        let pressure = classify_memory_pressure(probe.free_fraction());
        let effective = effective_admission_limit(limit, pressure);
        for slot in 0..effective {
            let path = root.join(format!("heavy-{slot}.lock"));
            match try_lock_slot(&path) {
                Ok(Some(slot)) => {
                    return Ok(HeavyCommandPermit {
                        _slot: slot,
                        queued_for: started.elapsed(),
                        limit,
                        memory_pressure: pressure,

View on GitHub (pinned to 0c42157ee5)

Solutions

Retry the heavy command once memory pressure drops or concurrent heavy commands finish (their lock files release).
Reduce concurrent heavy commands so slots free up faster.
Free host memory or raise the configured admission limit if the workload justifies it.
If the cancellation was a timeout, raise the caller's timeout to outlast the admission queue.

Defensive patterns

Strategy: retry

Validate before calling

// Preflight: skip queueing when the host is under critical pressure.
let pressure = classify_memory_pressure(memory_probe.free_fraction());
if pressure == MemoryPressure::Critical {
    return Err(anyhow::anyhow!("host under critical memory pressure; defer heavy command"));
}
acquire_heavy_command_permit(&root, limit, Some(&token), &probe).await?;

Try / catch

match acquire_heavy_command_permit(&root, limit, Some(&cancel_token), &probe).await {
    Ok(permit) => permit,
    Err(e) if e.to_string().contains("canceled while queued") => {
        // Admission queue outlasted the caller's patience: back off, let pressure
        // drain or slots free, then re-acquire with the same cancellation budget.
        tokio::time::sleep(Duration::from_secs(30)).await;
        acquire_heavy_command_permit(&root, limit, Some(&new_token), &probe).await?
    }
    Err(e) => return Err(e),
};

Prevention

Check host memory pressure before enqueueing heavy commands; defer when free memory is low.
Cap concurrent heavy commands at or below the configured admission limit so queueing is rare.
Size upstream timeouts to outlast worst-case admission waits instead of cancel-then-retry loops.

When it happens

Trigger: A heavy command waits for admission because all slot files are locked or memory pressure (measured via MemoryProbe.free_fraction) has reduced the effective limit to zero, and while waiting the caller cancels — e.g. session shutdown, tool-call abort, or a timeout token trips.

Common situations: Host under sustained memory pressure from several concurrent heavy link-graph builds; a user cancels a long-queued build; CI box memory pressure makes admission wait minutes and an upstream timeout cancels first.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/6b3be317482bede7. Report an issue: GitHub.