windmill-labs/windmill · info

S3 pull was canceled

Error message

S3 pull was canceled

What it means

For non-pro workspaces on enterprise+parquet builds, handle_python_reqs tries to restore the venv from an S3 tarball instead of installing packages. The pull is wrapped in tokio::select! with kill_rx; if the job is canceled while the S3 download is in flight, the code returns "S3 pull was canceled". The cancellation wins the race against pull_from_tar.

Source

Thrown at backend/windmill-worker/src/python_executor.rs:2904

            // Double-checked: another job (this process or another sharing the
            // mount) may have installed this exact dep while we waited on the
            // locks. Reuse it instead of reinstalling.
            if metadata(format!("{venv_p}/.valid.windmill")).await.is_ok() {
                print_success(
                    false, false, &job_id, &w_id, &req, req_tl, counter_arc,
                    total_to_install, start, &conn,
                )
                .await;
                pids.lock().await.get_mut(i).and_then(|e| e.take());
                return Ok(());
            }

            #[cfg(all(feature = "enterprise", feature = "parquet"))]
            if is_not_pro {
                if let Some(os) = windmill_object_store::get_object_store().await {
                    tokio::select! {
                        // Cancel was called on the job
                        _ = kill_rx.recv() => return Err(Error::from(anyhow::anyhow!("S3 pull was canceled"))),
                        pull = pull_from_tar(os, venv_p.clone(), py_version.to_cache_dir_top_level(false), None, false) => {
                            if let Err(e) = pull {
                                tracing::info!(
                                    workspace_id = %w_id,
                                    "No tarball was found for {venv_p} on S3 or different problem occurred {job_id}:\n{e}",
                                );
                            } else if let Err(verify_err) = verify_wheel_record(&venv_p).await {
                                // The object-store tar extracted cleanly but the resulting
                                // directory is missing files referenced by the wheel RECORD.
                                // Wipe the broken cache entry and fall through to a fresh
                                // local install rather than treating it as authoritative.
                                tracing::warn!(
                                    workspace_id = %w_id,
                                    job_id = %job_id,
                                    "Object-store cache for {venv_p} failed wheel RECORD verification, will reinstall locally: {verify_err}"
                                );
                                if let Err(rm_err) = tokio::fs::remove_dir_all(&venv_p).await {
                                    tracing::warn!(

View on GitHub (pinned to e474e8803c)

Solutions

1. No action needed for an intentional cancel — rerun the job; the tarball remains cached if the pull completed.
2. Speed up restores (smaller venv, re-upload a pruned tarball) so downloads finish before timeouts.
3. If cancels are spurious, check flow/step timeout configuration and increase it.
4. Verify object storage connectivity/latency; slow S3 access widens the cancellation window.

try {
  return await runPythonStep();
} catch (e) {
  if (String(e.message).includes('S3 pull was canceled')) {
    return await runPythonStep(); // tarball may now be cached locally
  }
  throw e;
}

When it happens

Trigger: A Python job on a non-pro workspace (enterprise worker with object storage configured) is canceled during the phase where its venv is being pulled from S3 object storage — user stop, flow cancellation, or job timeout hitting mid-download.

Common situations: Canceling a slow first run whose venv tarball is large or downloading over a slow S3 link; a flow step exceeding its timeout during venv restore; stopping a duplicated flow run while several steps restore the same venv.

Related errors

• install of {venv_p} canceled while waiting for venv lock
• install of {venv_p} canceled while waiting for venv file loc
• Could not generate DuckDB S3 connection settings from the pr
• Could not generate Polars S3 connection settings from the pr
• Could not generate Boto3 S3 connection settings from the pro

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/65611c5463206845. Report an issue: GitHub.