windmill-labs/windmill · info

install of {venv_p} canceled while waiting for venv file loc

Error message

install of {venv_p} canceled while waiting for venv file lock

What it means

After acquiring the in-process venv lock, handle_python_reqs must acquire the cross-process file lock guarding the venv directory. It polls in a 200ms loop, and each iteration uses tokio::select! against kill_rx; if the job is canceled while still waiting for the file lock, it removes its tracked pid and returns "install of {venv_p} canceled while waiting for venv file lock". Another OS process (a different worker or a stale lock holder) held the file lock when cancellation arrived.

Source

Thrown at backend/windmill-worker/src/python_executor.rs:2859

                        const MAX_WAIT: std::time::Duration = std::time::Duration::from_secs(300);
                        let waited_since = std::time::Instant::now();
                        loop {
                            match f.try_lock_exclusive() {
                                Ok(true) => break Some(f),
                                // Another holder has the lock.
                                Ok(false) => {
                                    if waited_since.elapsed() >= MAX_WAIT {
                                        tracing::warn!(
                                            workspace_id = %w_id,
                                            "venv install lock {lock_path} still held after {}s, proceeding without cross-process install lock",
                                            MAX_WAIT.as_secs()
                                        );
                                        break Some(f);
                                    }
                                    tokio::select! {
                                        _ = kill_rx.recv() => {
                                            pids.lock().await.get_mut(i).and_then(|e| e.take());
                                            return Err(Error::from(anyhow::anyhow!(
                                                "install of {venv_p} canceled while waiting for venv file lock"
                                            )));
                                        }
                                        _ = tokio::time::sleep(std::time::Duration::from_millis(200)) => {}
                                    }
                                }
                                Err(e) => {
                                    tracing::warn!(
                                        workspace_id = %w_id,
                                        "could not lock {lock_path}, proceeding without cross-process install lock: {e}"
                                    );
                                    break Some(f);
                                }
                            }
                        }
                    }
                    Err(e) => {
                        tracing::warn!(

View on GitHub (pinned to e474e8803c)

Solutions

1. Rerun the job if cancellation was intentional.
2. Check for a stale venv lock file (in the venv/cache dir) left by a crashed process and remove it after confirming no uv/pip process is alive.
3. Reduce install duration (trim requirements, pin wheels) so lock wait times stay below job timeouts.
4. Ensure only one worker at a time needs a given cold venv, or pre-warm the venv to skip installs entirely.

try {
  return await runPythonStep();
} catch (e) {
  if (String(e.message).includes('canceled while waiting for venv file lock')) {
    return await runPythonStep(); // lock holder likely finished installing
  }
  throw e;
}

When it happens

Trigger: A Python dependency install is canceled (user stop, flow cancellation, timeout) while a cross-process file lock on the venv directory is held by another worker process or a leftover lock from a killed process, so the poll loop never acquires it before kill_rx fires.

Common situations: Two workers installing the same venv concurrently while an operator cancels one; a crashed worker left a stale lock file and subsequent jobs are canceled while polling; long installs (large requirements) holding the lock past a job timeout.

Related errors

• install of {venv_p} canceled while waiting for venv lock
• S3 pull was canceled
• uv pip install was canceled
• could not read the run of job ${id}: ${res.status} ${await r
• could not record run ${experiment_id}: ${res.status} ${await

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/d98d2a031a07a707. Report an issue: GitHub.