windmill-labs/windmill · error · Error

could not read the run of job ${id}: ${res.status} ${await r

Error message

could not read the run of job ${id}: ${res.status} ${await res.text()}

What it means

Windmill verifies the installed wheel's RECORD file after pip-installing each requirement into the job's virtualenv. If the RECORD metadata does not match (the wheel file set differs from what the metadata declares), the venv is considered corrupt and this error is thrown after attempting to remove the broken install directory.

Source

Thrown at backend/windmill-api/src/ai_evals/run.rs:42

    #[serde(skip_serializing_if = "Option::is_none")]
    expected: Option<Box<RawValue>>,
}

/// Assembles the payload the scorers read.
///
/// A step rather than an input transform: every tool call is enriched with the arguments, result,
/// status and duration of the job that ran it, none of which the flow can see.
const PAYLOAD_SCRIPT: &str = r#"//native
// Generated by Windmill: reads the run this iteration answered.
export async function main() {
  const id = process.env.WM_FLOW_JOB_ID
  const base = process.env.BASE_URL || process.env.BASE_INTERNAL_URL
  const res = await fetch(
    `${base}/api/w/${process.env.WM_WORKSPACE}/ai_evals/run_payload?job_id=${id}`,
    { headers: { Authorization: `Bearer ${process.env.WM_TOKEN}` } }
  )
  if (!res.ok) {
    throw new Error(`could not read the run of job ${id}: ${res.status} ${await res.text()}`)
  }
  return await res.json()
}
"#;

fn payload_module() -> serde_json::Value {
    serde_json::json!({
        "id": PAYLOAD_NODE_ID,
        "summary": "Assemble the run the scorers read",
        "value": {
            "type": "rawscript",
            // `bunnative` (tag `nativets`), matching the `//native` the script carries. That tag
            // belongs to the `native` worker group rather than the default one, so a queued
            // iteration never starts when nothing serves it.
            "language": "bunnative",
            "content": PAYLOAD_SCRIPT,
            "lock": EMPTY_BUN_LOCK,
            "input_transforms": {}

View on GitHub (pinned to e474e8803c)

Solutions

1. Retry the job — the code removes the broken install dir and a fresh run reinstalls the wheel from scratch
2. Clear the pip/windmill wheel cache on the worker and retry to force a clean re-download
3. Check disk space on the worker host (df -h) and free space if the disk was full
4. Use a reliable PyPI index/mirror (check PIP_INDEX_URL) and verify network stability
5. Pin the dependency to an exact version to avoid a newly-published (possibly corrupt-in-cache) release

Example fix

// before (flaky mirror)
PIP_INDEX_URL=https://internal-mirror.example.com/simple
// after (official index or verified mirror)
PIP_INDEX_URL=https://pypi.org/simple

// Pre-check wheels/caches before install (worker-side)
import { execSync } from 'child_process';
function validatePipReqs(reqs) {
  if (!Array.isArray(reqs) || reqs.length === 0) throw new Error('no requirements');
  for (const r of reqs) {
    if (/[\s;]/.test(r)) throw new Error(`suspicious requirement: ${r}`);
  }
  execSync('df -h .', { stdio: 'inherit' }); // ensure disk headroom
}
validatePipReqs(['pandas==2.2.2']);

function isWellFormedReq(req) {
  return typeof req === 'string' && /^[A-Za-z0-9_.\-]+(==[^\s;]+)?$/.test(req);
}

for (let attempt = 1; attempt <= 3; attempt++) {
  try { await installReqs(reqs); break; }
  catch (e) {
    if (!/wheel RECORD verification failed/.test(e.message) || attempt === 3) throw e;
    clearPipCache(); // force clean re-download
  }
}

When it happens

Trigger: handle_python_reqs installs a wheel whose RECORD entries do not match the files actually extracted — typically a corrupted/truncated wheel download, a partially-written install (e.g. killed mid-install), or disk full during extraction.

Common situations: Unstable network causing truncated wheel downloads; worker killed (OOM/restart) during pip install leaving a partial install; flaky PyPI mirror or proxy serving corrupt artifacts; disk quota exhaustion in the shared pip cache.

Related errors

• could not record run ${experiment_id}: ${res.status} ${await
• file path escapes the build directory: ${rel}
• ${what} failed:\n${output}
• bundle failed:\n${attempt.output}
• duplicate MCP tool name(s): {', '.join(duplicates)}

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/4a1298eec680700c. Report an issue: GitHub.