{"record":{"id":"b9701582928b4e22","repo":"windmill-labs/windmill","slug":"install-of-venv-p-canceled-while-waiting-for-ven","errorCode":null,"errorMessage":"install of {venv_p} canceled while waiting for venv lock","messagePattern":"install of (.+?) canceled while waiting for venv lock","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"info","filePath":"backend/windmill-worker/src/python_executor.rs","lineNumber":2816,"sourceCode":"\n            tracing::info!(\n                workspace_id = %w_id,\n                job_id = %job_id,\n                // is_ok = out,\n                \"started thread to install wheel {}\",\n                venv_p\n            );\n\n            let start = std::time::Instant::now();\n\n            // Lock the shared target dir (see PY_INSTALL_LOCKS). In-process lock\n            // first; only one task per dir then contends the cross-process file\n            // lock below. Both guards drop on every return path.\n            let venv_lock = get_venv_install_lock(&venv_p).await;\n            let _venv_guard = tokio::select! {\n                _ = kill_rx.recv() => {\n                    pids.lock().await.get_mut(i).and_then(|e| e.take());\n                    return Err(Error::from(anyhow::anyhow!(\n                        \"install of {venv_p} canceled while waiting for venv lock\"\n                    )));\n                }\n                guard = venv_lock.lock_owned() => guard,\n            };\n\n            // Cross-process advisory lock. Best-effort: if the filesystem doesn't\n            // support locking we log and proceed — verify_wheel_record + job retry\n            // still guard correctness, just without the dedup. Cross-platform\n            // (flock on unix, LockFileEx on windows) so agents sharing a wheel-cache\n            // dir on a Windows host serialize just as they do on unix.\n            let _venv_file_lock: Option<std::fs::File> = {\n                use fs4::fs_std::FileExt;\n                let lock_path = format!(\"{venv_p}.lock\");\n                if let Some(parent) = std::path::Path::new(&lock_path).parent() {\n                    let _ = std::fs::create_dir_all(parent);\n                }\n                match std::fs::OpenOptions::new().create(true).write(true).open(&lock_path) {","sourceCodeStart":2798,"sourceCodeEnd":2834,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/backend/windmill-worker/src/python_executor.rs#L2798-L2834","documentation":"Before installing a Python venv, handle_python_reqs acquires an in-process venv lock and then a cross-process file lock. While waiting for the in-process venv lock, it listens on the job's kill_rx channel; if the job is canceled first, it removes its tracked pid and returns \"install of {venv_p} canceled while waiting for venv lock\". This is a deliberate cancellation path, not a bug in the install itself.","triggerScenarios":"A Python job with unmet dependencies is canceled (user cancellation, flow stop, timeout, worker shutdown) at the exact moment another job/task in the same worker process already holds the venv's install lock, so the canceled job wins the select! race.","commonSituations":"Stopping a flow that launches several Python steps needing the same fresh venv; canceling a long-queued script just as its dependencies started installing; job timeout expiring during a contended install; multiple concurrent first-runs of scripts sharing one requirements set.","solutions":["No action needed if the cancellation was intentional — rerun the job.","Reduce contention: pre-install the dependencies by running the script once to completion, so later runs skip the install path.","Avoid launching many Python jobs with identical unmet deps simultaneously; stagger them or give them distinct deduplicated requirement sets.","Check who holds the venv lock (another long-running install) and wait for it to finish before resubmitting."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// in the flow step, retry once on cancellation\ntry {\n  return await runPythonStep();\n} catch (e) {\n  if (String(e.message).includes('canceled while waiting for venv lock')) {\n    return await runPythonStep(); // venv is likely warm now\n  }\n  throw e;\n}","preventionTips":["Pre-warm venvs by running dependency scripts once before launching parallel workloads","Avoid launching many jobs sharing an unmet requirement set at the same instant","Set generous timeouts for first runs that include dependency installation"],"tags":["python","cancellation","locking","venv"],"backgroundTag":"operation-canceled","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}