windmill-labs/windmill · info

install of {venv_p} canceled while waiting for venv lock

Error message

install of {venv_p} canceled while waiting for venv lock

What it means

Before installing a Python venv, handle_python_reqs acquires an in-process venv lock and then a cross-process file lock. While waiting for the in-process venv lock, it listens on the job's kill_rx channel; if the job is canceled first, it removes its tracked pid and returns "install of {venv_p} canceled while waiting for venv lock". This is a deliberate cancellation path, not a bug in the install itself.

Source

Thrown at backend/windmill-worker/src/python_executor.rs:2816

            tracing::info!(
                workspace_id = %w_id,
                job_id = %job_id,
                // is_ok = out,
                "started thread to install wheel {}",
                venv_p
            );

            let start = std::time::Instant::now();

            // Lock the shared target dir (see PY_INSTALL_LOCKS). In-process lock
            // first; only one task per dir then contends the cross-process file
            // lock below. Both guards drop on every return path.
            let venv_lock = get_venv_install_lock(&venv_p).await;
            let _venv_guard = tokio::select! {
                _ = kill_rx.recv() => {
                    pids.lock().await.get_mut(i).and_then(|e| e.take());
                    return Err(Error::from(anyhow::anyhow!(
                        "install of {venv_p} canceled while waiting for venv lock"
                    )));
                }
                guard = venv_lock.lock_owned() => guard,
            };

            // Cross-process advisory lock. Best-effort: if the filesystem doesn't
            // support locking we log and proceed — verify_wheel_record + job retry
            // still guard correctness, just without the dedup. Cross-platform
            // (flock on unix, LockFileEx on windows) so agents sharing a wheel-cache
            // dir on a Windows host serialize just as they do on unix.
            let _venv_file_lock: Option<std::fs::File> = {
                use fs4::fs_std::FileExt;
                let lock_path = format!("{venv_p}.lock");
                if let Some(parent) = std::path::Path::new(&lock_path).parent() {
                    let _ = std::fs::create_dir_all(parent);
                }
                match std::fs::OpenOptions::new().create(true).write(true).open(&lock_path) {

View on GitHub (pinned to e474e8803c)

Solutions

1. No action needed if the cancellation was intentional — rerun the job.
2. Reduce contention: pre-install the dependencies by running the script once to completion, so later runs skip the install path.
3. Avoid launching many Python jobs with identical unmet deps simultaneously; stagger them or give them distinct deduplicated requirement sets.
4. Check who holds the venv lock (another long-running install) and wait for it to finish before resubmitting.

// in the flow step, retry once on cancellation
try {
  return await runPythonStep();
} catch (e) {
  if (String(e.message).includes('canceled while waiting for venv lock')) {
    return await runPythonStep(); // venv is likely warm now
  }
  throw e;
}

When it happens

Trigger: A Python job with unmet dependencies is canceled (user cancellation, flow stop, timeout, worker shutdown) at the exact moment another job/task in the same worker process already holds the venv's install lock, so the canceled job wins the select! race.

Common situations: Stopping a flow that launches several Python steps needing the same fresh venv; canceling a long-queued script just as its dependencies started installing; job timeout expiring during a contended install; multiple concurrent first-runs of scripts sharing one requirements set.

Related errors

• install of {venv_p} canceled while waiting for venv file loc
• S3 pull was canceled
• uv pip install was canceled
• could not read the run of job ${id}: ${res.status} ${await r
• could not record run ${experiment_id}: ${res.status} ${await

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/b9701582928b4e22. Report an issue: GitHub.