{"record":{"id":"d2fc7b0a85d86801","repo":"windmill-labs/windmill","slug":"could-not-record-run-experiment-id-res-statu","errorCode":null,"errorMessage":"could not record run ${experiment_id}: ${res.status} ${await res.text()}","messagePattern":"could not record run (.+?): (.+?) (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"backend/windmill-api/src/ai_evals/run.rs","lineNumber":82,"sourceCode":"\n/// Node id of the step that records what the run produced.\nconst COLLECT_NODE_ID: &str = \"collect\";\n\n/// Copies the run's answers and scores into its own rows, from inside the run.\n///\n/// The tables know nothing about the flow, so without this a run started and left is only ever\n/// recorded by someone looking at it — after its jobs have been retained away, there is nothing\n/// left to record.\nconst COLLECT_SCRIPT: &str = r#\"//native\n// Generated by Windmill: records what this run produced, so it outlives the jobs that produced it.\nexport async function main(experiment_id: string) {\n  const base = process.env.BASE_URL || process.env.BASE_INTERNAL_URL\n  const res = await fetch(\n    `${base}/api/w/${process.env.WM_WORKSPACE}/ai_evals/experiments/collect?id=${experiment_id}`,\n    { method: 'POST', headers: { Authorization: `Bearer ${process.env.WM_TOKEN}` } }\n  )\n  if (!res.ok) {\n    throw new Error(`could not record run ${experiment_id}: ${res.status} ${await res.text()}`)\n  }\n  return await res.json()\n}\n\"#;\n\nfn collect_module(experiment_id: Uuid) -> serde_json::Value {\n    serde_json::json!({\n        \"id\": COLLECT_NODE_ID,\n        \"summary\": \"Record what the run produced\",\n        // Bookkeeping, so it does not decide whether the run succeeded. What it would have written\n        // is written again by the first read of the run.\n        \"continue_on_error\": true,\n        \"value\": {\n            \"type\": \"rawscript\",\n            \"language\": \"bunnative\",\n            \"content\": COLLECT_SCRIPT,\n            \"lock\": EMPTY_BUN_LOCK,\n            \"input_transforms\": {","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/backend/windmill-api/src/ai_evals/run.rs#L64-L100","documentation":"After spawning parallel pip installs (one per requirement/venv), handle_python_reqs joins each task handle. This fallback error is produced when a join itself fails or the handle returns None — i.e. the install task panicked or was cancelled rather than completing with a normal result.","triggerScenarios":"The spawned install task panics (e.g. a bug in install plumbing) or is cancelled, so handle.await yields Err/None; the code substitutes this generic message instead of a real cause.","commonSituations":"Worker under extreme memory pressure causing panics in child-process handling; tokio runtime shutdown mid-install; a panic inside the per-requirement install future (check worker logs for the panic backtrace just before this message).","solutions":["Inspect worker logs immediately preceding this error for a panic backtrace — it carries the real root cause","Reduce parallelism of dependency installs (fewer concurrent jobs/venvs) to rule out resource exhaustion","Retry the deployment; transient runtime cancellation resolves on a clean rerun","Restart the worker to clear any degraded state and retry","If reproducible, capture the panic and file a bug — this message masks the underlying cause"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  await installReqs(reqs);\n} catch (e) {\n  if (/Problem by joining handle/.test(e.message)) {\n    // real cause is a panic upstream — surface worker logs, retry once\n    logger.error('install task panicked/cancelled; check worker logs', { reqs });\n    await installReqs(reqs); // retry once\n  } else throw e;\n}","preventionTips":["Ensure workers have adequate memory to avoid panics under pressure","Keep worker runtime (tokio) healthy; avoid abrupt runtime shutdowns","Always read worker logs around this error — it masks the true panic","Reduce concurrent install parallelism on small workers"],"tags":["python","tokio","task-panic","dependency-installation"],"backgroundTag":"task-join-failed","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}