What causes "could not record run ${experiment_id}: ${res.status} ${await res.text()}" in windmill-labs/windmill?

Inspect worker logs immediately preceding this error for a panic backtrace — it carries the real root cause

windmill-labs/windmill · error · Error

could not record run ${experiment_id}: ${res.status} ${await

Error message

could not record run ${experiment_id}: ${res.status} ${await res.text()}

What it means

After spawning parallel pip installs (one per requirement/venv), handle_python_reqs joins each task handle. This fallback error is produced when a join itself fails or the handle returns None — i.e. the install task panicked or was cancelled rather than completing with a normal result.

Source

Thrown at backend/windmill-api/src/ai_evals/run.rs:82

/// Node id of the step that records what the run produced.
const COLLECT_NODE_ID: &str = "collect";

/// Copies the run's answers and scores into its own rows, from inside the run.
///
/// The tables know nothing about the flow, so without this a run started and left is only ever
/// recorded by someone looking at it — after its jobs have been retained away, there is nothing
/// left to record.
const COLLECT_SCRIPT: &str = r#"//native
// Generated by Windmill: records what this run produced, so it outlives the jobs that produced it.
export async function main(experiment_id: string) {
  const base = process.env.BASE_URL || process.env.BASE_INTERNAL_URL
  const res = await fetch(
    `${base}/api/w/${process.env.WM_WORKSPACE}/ai_evals/experiments/collect?id=${experiment_id}`,
    { method: 'POST', headers: { Authorization: `Bearer ${process.env.WM_TOKEN}` } }
  )
  if (!res.ok) {
    throw new Error(`could not record run ${experiment_id}: ${res.status} ${await res.text()}`)
  }
  return await res.json()
}
"#;

fn collect_module(experiment_id: Uuid) -> serde_json::Value {
    serde_json::json!({
        "id": COLLECT_NODE_ID,
        "summary": "Record what the run produced",
        // Bookkeeping, so it does not decide whether the run succeeded. What it would have written
        // is written again by the first read of the run.
        "continue_on_error": true,
        "value": {
            "type": "rawscript",
            "language": "bunnative",
            "content": COLLECT_SCRIPT,
            "lock": EMPTY_BUN_LOCK,
            "input_transforms": {

View on GitHub (pinned to e474e8803c)

Solutions

Inspect worker logs immediately preceding this error for a panic backtrace — it carries the real root cause
Reduce parallelism of dependency installs (fewer concurrent jobs/venvs) to rule out resource exhaustion
Retry the deployment; transient runtime cancellation resolves on a clean rerun
Restart the worker to clear any degraded state and retry
If reproducible, capture the panic and file a bug — this message masks the underlying cause

Defensive patterns

Strategy: try-catch

Try / catch

try {
  await installReqs(reqs);
} catch (e) {
  if (/Problem by joining handle/.test(e.message)) {
    // real cause is a panic upstream — surface worker logs, retry once
    logger.error('install task panicked/cancelled; check worker logs', { reqs });
    await installReqs(reqs); // retry once
  } else throw e;
}

Prevention

Ensure workers have adequate memory to avoid panics under pressure
Keep worker runtime (tokio) healthy; avoid abrupt runtime shutdowns
Always read worker logs around this error — it masks the true panic
Reduce concurrent install parallelism on small workers

When it happens

Trigger: The spawned install task panics (e.g. a bug in install plumbing) or is cancelled, so handle.await yields Err/None; the code substitutes this generic message instead of a real cause.

Common situations: Worker under extreme memory pressure causing panics in child-process handling; tokio runtime shutdown mid-install; a panic inside the per-requirement install future (check worker logs for the panic backtrace just before this message).

Related errors

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/d2fc7b0a85d86801. Report an issue: GitHub.