{"record":{"id":"d98d2a031a07a707","repo":"windmill-labs/windmill","slug":"install-of-venv-p-canceled-while-waiting-for-ven-d98d2a","errorCode":null,"errorMessage":"install of {venv_p} canceled while waiting for venv file lock","messagePattern":"install of (.+?) canceled while waiting for venv file lock","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"info","filePath":"backend/windmill-worker/src/python_executor.rs","lineNumber":2859,"sourceCode":"                        const MAX_WAIT: std::time::Duration = std::time::Duration::from_secs(300);\n                        let waited_since = std::time::Instant::now();\n                        loop {\n                            match f.try_lock_exclusive() {\n                                Ok(true) => break Some(f),\n                                // Another holder has the lock.\n                                Ok(false) => {\n                                    if waited_since.elapsed() >= MAX_WAIT {\n                                        tracing::warn!(\n                                            workspace_id = %w_id,\n                                            \"venv install lock {lock_path} still held after {}s, proceeding without cross-process install lock\",\n                                            MAX_WAIT.as_secs()\n                                        );\n                                        break Some(f);\n                                    }\n                                    tokio::select! {\n                                        _ = kill_rx.recv() => {\n                                            pids.lock().await.get_mut(i).and_then(|e| e.take());\n                                            return Err(Error::from(anyhow::anyhow!(\n                                                \"install of {venv_p} canceled while waiting for venv file lock\"\n                                            )));\n                                        }\n                                        _ = tokio::time::sleep(std::time::Duration::from_millis(200)) => {}\n                                    }\n                                }\n                                Err(e) => {\n                                    tracing::warn!(\n                                        workspace_id = %w_id,\n                                        \"could not lock {lock_path}, proceeding without cross-process install lock: {e}\"\n                                    );\n                                    break Some(f);\n                                }\n                            }\n                        }\n                    }\n                    Err(e) => {\n                        tracing::warn!(","sourceCodeStart":2841,"sourceCodeEnd":2877,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/backend/windmill-worker/src/python_executor.rs#L2841-L2877","documentation":"After acquiring the in-process venv lock, handle_python_reqs must acquire the cross-process file lock guarding the venv directory. It polls in a 200ms loop, and each iteration uses tokio::select! against kill_rx; if the job is canceled while still waiting for the file lock, it removes its tracked pid and returns \"install of {venv_p} canceled while waiting for venv file lock\". Another OS process (a different worker or a stale lock holder) held the file lock when cancellation arrived.","triggerScenarios":"A Python dependency install is canceled (user stop, flow cancellation, timeout) while a cross-process file lock on the venv directory is held by another worker process or a leftover lock from a killed process, so the poll loop never acquires it before kill_rx fires.","commonSituations":"Two workers installing the same venv concurrently while an operator cancels one; a crashed worker left a stale lock file and subsequent jobs are canceled while polling; long installs (large requirements) holding the lock past a job timeout.","solutions":["Rerun the job if cancellation was intentional.","Check for a stale venv lock file (in the venv/cache dir) left by a crashed process and remove it after confirming no uv/pip process is alive.","Reduce install duration (trim requirements, pin wheels) so lock wait times stay below job timeouts.","Ensure only one worker at a time needs a given cold venv, or pre-warm the venv to skip installs entirely."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return await runPythonStep();\n} catch (e) {\n  if (String(e.message).includes('canceled while waiting for venv file lock')) {\n    return await runPythonStep(); // lock holder likely finished installing\n  }\n  throw e;\n}","preventionTips":["Check for and clean stale venv lock files after worker crashes","Keep requirement sets small/pinned so cross-process lock waits stay short","Pre-warm shared venvs so jobs skip the install-and-lock path entirely"],"tags":["python","cancellation","file-lock","venv"],"backgroundTag":"operation-canceled","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}