{"record":{"id":"4a1298eec680700c","repo":"windmill-labs/windmill","slug":"could-not-read-the-run-of-job-id-res-status","errorCode":null,"errorMessage":"could not read the run of job ${id}: ${res.status} ${await res.text()}","messagePattern":"could not read the run of job (.+?): (.+?) (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"backend/windmill-api/src/ai_evals/run.rs","lineNumber":42,"sourceCode":"    #[serde(skip_serializing_if = \"Option::is_none\")]\n    expected: Option<Box<RawValue>>,\n}\n\n/// Assembles the payload the scorers read.\n///\n/// A step rather than an input transform: every tool call is enriched with the arguments, result,\n/// status and duration of the job that ran it, none of which the flow can see.\nconst PAYLOAD_SCRIPT: &str = r#\"//native\n// Generated by Windmill: reads the run this iteration answered.\nexport async function main() {\n  const id = process.env.WM_FLOW_JOB_ID\n  const base = process.env.BASE_URL || process.env.BASE_INTERNAL_URL\n  const res = await fetch(\n    `${base}/api/w/${process.env.WM_WORKSPACE}/ai_evals/run_payload?job_id=${id}`,\n    { headers: { Authorization: `Bearer ${process.env.WM_TOKEN}` } }\n  )\n  if (!res.ok) {\n    throw new Error(`could not read the run of job ${id}: ${res.status} ${await res.text()}`)\n  }\n  return await res.json()\n}\n\"#;\n\nfn payload_module() -> serde_json::Value {\n    serde_json::json!({\n        \"id\": PAYLOAD_NODE_ID,\n        \"summary\": \"Assemble the run the scorers read\",\n        \"value\": {\n            \"type\": \"rawscript\",\n            // `bunnative` (tag `nativets`), matching the `//native` the script carries. That tag\n            // belongs to the `native` worker group rather than the default one, so a queued\n            // iteration never starts when nothing serves it.\n            \"language\": \"bunnative\",\n            \"content\": PAYLOAD_SCRIPT,\n            \"lock\": EMPTY_BUN_LOCK,\n            \"input_transforms\": {}","sourceCodeStart":24,"sourceCodeEnd":60,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/backend/windmill-api/src/ai_evals/run.rs#L24-L60","documentation":"Windmill verifies the installed wheel's RECORD file after pip-installing each requirement into the job's virtualenv. If the RECORD metadata does not match (the wheel file set differs from what the metadata declares), the venv is considered corrupt and this error is thrown after attempting to remove the broken install directory.","triggerScenarios":"handle_python_reqs installs a wheel whose RECORD entries do not match the files actually extracted — typically a corrupted/truncated wheel download, a partially-written install (e.g. killed mid-install), or disk full during extraction.","commonSituations":"Unstable network causing truncated wheel downloads; worker killed (OOM/restart) during pip install leaving a partial install; flaky PyPI mirror or proxy serving corrupt artifacts; disk quota exhaustion in the shared pip cache.","solutions":["Retry the job — the code removes the broken install dir and a fresh run reinstalls the wheel from scratch","Clear the pip/windmill wheel cache on the worker and retry to force a clean re-download","Check disk space on the worker host (df -h) and free space if the disk was full","Use a reliable PyPI index/mirror (check PIP_INDEX_URL) and verify network stability","Pin the dependency to an exact version to avoid a newly-published (possibly corrupt-in-cache) release"],"exampleFix":"// before (flaky mirror)\nPIP_INDEX_URL=https://internal-mirror.example.com/simple\n// after (official index or verified mirror)\nPIP_INDEX_URL=https://pypi.org/simple","handlingStrategy":"retry","validationCode":"// Pre-check wheels/caches before install (worker-side)\nimport { execSync } from 'child_process';\nfunction validatePipReqs(reqs) {\n  if (!Array.isArray(reqs) || reqs.length === 0) throw new Error('no requirements');\n  for (const r of reqs) {\n    if (/[\\s;]/.test(r)) throw new Error(`suspicious requirement: ${r}`);\n  }\n  execSync('df -h .', { stdio: 'inherit' }); // ensure disk headroom\n}\nvalidatePipReqs(['pandas==2.2.2']);","typeGuard":"function isWellFormedReq(req) {\n  return typeof req === 'string' && /^[A-Za-z0-9_.\\-]+(==[^\\s;]+)?$/.test(req);\n}","tryCatchPattern":"for (let attempt = 1; attempt <= 3; attempt++) {\n  try { await installReqs(reqs); break; }\n  catch (e) {\n    if (!/wheel RECORD verification failed/.test(e.message) || attempt === 3) throw e;\n    clearPipCache(); // force clean re-download\n  }\n}","preventionTips":["Use a reliable PyPI index or verified internal mirror","Monitor worker disk space; keep pip cache on a roomy volume","Avoid killing workers mid-install; use graceful shutdown","Pin dependency versions so failed wheels are easy to isolate"],"tags":["python","pip","dependency-installation","corruption"],"backgroundTag":"wheel-record-verification-failed","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}