{"record":{"id":"65611c5463206845","repo":"windmill-labs/windmill","slug":"s3-pull-was-canceled","errorCode":null,"errorMessage":"S3 pull was canceled","messagePattern":"S3 pull was canceled","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"info","filePath":"backend/windmill-worker/src/python_executor.rs","lineNumber":2904,"sourceCode":"            // Double-checked: another job (this process or another sharing the\n            // mount) may have installed this exact dep while we waited on the\n            // locks. Reuse it instead of reinstalling.\n            if metadata(format!(\"{venv_p}/.valid.windmill\")).await.is_ok() {\n                print_success(\n                    false, false, &job_id, &w_id, &req, req_tl, counter_arc,\n                    total_to_install, start, &conn,\n                )\n                .await;\n                pids.lock().await.get_mut(i).and_then(|e| e.take());\n                return Ok(());\n            }\n\n            #[cfg(all(feature = \"enterprise\", feature = \"parquet\"))]\n            if is_not_pro {\n                if let Some(os) = windmill_object_store::get_object_store().await {\n                    tokio::select! {\n                        // Cancel was called on the job\n                        _ = kill_rx.recv() => return Err(Error::from(anyhow::anyhow!(\"S3 pull was canceled\"))),\n                        pull = pull_from_tar(os, venv_p.clone(), py_version.to_cache_dir_top_level(false), None, false) => {\n                            if let Err(e) = pull {\n                                tracing::info!(\n                                    workspace_id = %w_id,\n                                    \"No tarball was found for {venv_p} on S3 or different problem occurred {job_id}:\\n{e}\",\n                                );\n                            } else if let Err(verify_err) = verify_wheel_record(&venv_p).await {\n                                // The object-store tar extracted cleanly but the resulting\n                                // directory is missing files referenced by the wheel RECORD.\n                                // Wipe the broken cache entry and fall through to a fresh\n                                // local install rather than treating it as authoritative.\n                                tracing::warn!(\n                                    workspace_id = %w_id,\n                                    job_id = %job_id,\n                                    \"Object-store cache for {venv_p} failed wheel RECORD verification, will reinstall locally: {verify_err}\"\n                                );\n                                if let Err(rm_err) = tokio::fs::remove_dir_all(&venv_p).await {\n                                    tracing::warn!(","sourceCodeStart":2886,"sourceCodeEnd":2922,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/backend/windmill-worker/src/python_executor.rs#L2886-L2922","documentation":"For non-pro workspaces on enterprise+parquet builds, handle_python_reqs tries to restore the venv from an S3 tarball instead of installing packages. The pull is wrapped in tokio::select! with kill_rx; if the job is canceled while the S3 download is in flight, the code returns \"S3 pull was canceled\". The cancellation wins the race against pull_from_tar.","triggerScenarios":"A Python job on a non-pro workspace (enterprise worker with object storage configured) is canceled during the phase where its venv is being pulled from S3 object storage — user stop, flow cancellation, or job timeout hitting mid-download.","commonSituations":"Canceling a slow first run whose venv tarball is large or downloading over a slow S3 link; a flow step exceeding its timeout during venv restore; stopping a duplicated flow run while several steps restore the same venv.","solutions":["No action needed for an intentional cancel — rerun the job; the tarball remains cached if the pull completed.","Speed up restores (smaller venv, re-upload a pruned tarball) so downloads finish before timeouts.","If cancels are spurious, check flow/step timeout configuration and increase it.","Verify object storage connectivity/latency; slow S3 access widens the cancellation window."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return await runPythonStep();\n} catch (e) {\n  if (String(e.message).includes('S3 pull was canceled')) {\n    return await runPythonStep(); // tarball may now be cached locally\n  }\n  throw e;\n}","preventionTips":["Increase flow/step timeouts if venv restores routinely exceed them","Keep S3 venv tarballs small and the object store close to workers","Monitor object-store latency; slow pulls widen the cancellation window"],"tags":["python","cancellation","s3","venv"],"backgroundTag":"operation-canceled","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}