{"record":{"id":"6b3be317482bede7","repo":"Hmbown/CodeWhale","slug":"heavy-command-canceled-while-queued-for-resource-a","errorCode":null,"errorMessage":"heavy command canceled while queued for resource admission","messagePattern":"heavy command canceled while queued for resource admission","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"warning","filePath":"crates/tui/src/tools/resource_admission.rs","lineNumber":188,"sourceCode":"    let probe = HostMemoryProbe;\n    acquire_heavy_command_permit_at(&root, limit, cancel, &probe)\n        .await\n        .map(Some)\n}\n\nasync fn acquire_heavy_command_permit_at(\n    root: &Path,\n    limit: usize,\n    cancel: Option<&CancellationToken>,\n    probe: &dyn MemoryProbe,\n) -> Result<HeavyCommandPermit> {\n    std::fs::create_dir_all(root)\n        .with_context(|| format!(\"creating resource admission directory {}\", root.display()))?;\n    let started = Instant::now();\n\n    loop {\n        if cancel.is_some_and(|token| token.is_cancelled()) {\n            return Err(anyhow!(\n                \"heavy command canceled while queued for resource admission\"\n            ));\n        }\n        // Re-measure each iteration: under memory pressure the effective limit\n        // tightens so a saturated host stops admitting new heavy link graphs\n        // (#4864 req 7). Critical pressure yields zero slots, so the command\n        // waits for recovery instead of snowballing.\n        let pressure = classify_memory_pressure(probe.free_fraction());\n        let effective = effective_admission_limit(limit, pressure);\n        for slot in 0..effective {\n            let path = root.join(format!(\"heavy-{slot}.lock\"));\n            match try_lock_slot(&path) {\n                Ok(Some(slot)) => {\n                    return Ok(HeavyCommandPermit {\n                        _slot: slot,\n                        queued_for: started.elapsed(),\n                        limit,\n                        memory_pressure: pressure,","sourceCodeStart":170,"sourceCodeEnd":206,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/0c42157ee52f9d55af2b506d71b46249910f77d3/crates/tui/src/tools/resource_admission.rs#L170-L206","documentation":"acquire_heavy_command_permit_at implements memory-aware admission for heavy commands: it loops trying to lock a slot file (heavy-N.lock) under root, where the effective slot count shrinks with host memory pressure (critical pressure yields zero slots). Each iteration checks the supplied CancellationToken; if the token fires while the command is still queued (no free slot at the current effective limit), this error is returned instead of a permit.","triggerScenarios":"A heavy command waits for admission because all slot files are locked or memory pressure (measured via MemoryProbe.free_fraction) has reduced the effective limit to zero, and while waiting the caller cancels — e.g. session shutdown, tool-call abort, or a timeout token trips.","commonSituations":"Host under sustained memory pressure from several concurrent heavy link-graph builds; a user cancels a long-queued build; CI box memory pressure makes admission wait minutes and an upstream timeout cancels first.","solutions":["Retry the heavy command once memory pressure drops or concurrent heavy commands finish (their lock files release).","Reduce concurrent heavy commands so slots free up faster.","Free host memory or raise the configured admission limit if the workload justifies it.","If the cancellation was a timeout, raise the caller's timeout to outlast the admission queue."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Preflight: skip queueing when the host is under critical pressure.\nlet pressure = classify_memory_pressure(memory_probe.free_fraction());\nif pressure == MemoryPressure::Critical {\n    return Err(anyhow::anyhow!(\"host under critical memory pressure; defer heavy command\"));\n}\nacquire_heavy_command_permit(&root, limit, Some(&token), &probe).await?;","typeGuard":null,"tryCatchPattern":"match acquire_heavy_command_permit(&root, limit, Some(&cancel_token), &probe).await {\n    Ok(permit) => permit,\n    Err(e) if e.to_string().contains(\"canceled while queued\") => {\n        // Admission queue outlasted the caller's patience: back off, let pressure\n        // drain or slots free, then re-acquire with the same cancellation budget.\n        tokio::time::sleep(Duration::from_secs(30)).await;\n        acquire_heavy_command_permit(&root, limit, Some(&new_token), &probe).await?\n    }\n    Err(e) => return Err(e),\n};","preventionTips":["Check host memory pressure before enqueueing heavy commands; defer when free memory is low.","Cap concurrent heavy commands at or below the configured admission limit so queueing is rare.","Size upstream timeouts to outlast worst-case admission waits instead of cancel-then-retry loops."],"tags":["resource-admission","memory-pressure","cancellation","concurrency","backpressure"],"backgroundTag":"operation-cancelled","analyzedSha":"0c42157ee52f9d55af2b506d71b46249910f77d3","analyzedAt":"2026-08-20T21:50:45.477Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}