{"record":{"id":"75b99573096e1c7e","repo":"windmill-labs/windmill","slug":"timed-out-after-math-round-cascade-job-timeout-m","errorCode":null,"errorMessage":"Timed out after ${Math.round(CASCADE_JOB_TIMEOUT_MS / 60000)}min waiting for job ${jobId} to finish","messagePattern":"Timed out after (.+?)min waiting for job (.+?) to finish","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"frontend/src/lib/components/assets/AssetGraph/cascadeRun.ts","lineNumber":56,"sourceCode":"export function makeWaitJobTerminal(\n\tworkspace: string\n): (jobId: string) => Promise<'success' | 'failure'> {\n\treturn async function waitJobTerminal(jobId: string): Promise<'success' | 'failure'> {\n\t\tconst deadline = Date.now() + CASCADE_JOB_TIMEOUT_MS\n\t\twhile (Date.now() < deadline) {\n\t\t\ttry {\n\t\t\t\tconst r = await JobService.getCompletedJobResultMaybe({\n\t\t\t\t\tworkspace,\n\t\t\t\t\tid: jobId,\n\t\t\t\t\tgetStarted: false\n\t\t\t\t})\n\t\t\t\tif (r.completed) return r.success ? 'success' : 'failure'\n\t\t\t} catch {\n\t\t\t\t// transient — retry on the next tick\n\t\t\t}\n\t\t\tawait new Promise((res) => setTimeout(res, CASCADE_POLL_INTERVAL_MS))\n\t\t}\n\t\tthrow new Error(\n\t\t\t`Timed out after ${Math.round(CASCADE_JOB_TIMEOUT_MS / 60000)}min waiting for job ${jobId} to finish`\n\t\t)\n\t}\n}\n\n// Build a per-script launch function. When `resolveLocal(path)` yields content,\n// the script runs as a preview of that local content (no deploy); otherwise it\n// runs the deployed version by path. Always passes `_wmill_skip_asset_dispatch`.\nexport function makeLaunch(opts: {\n\tworkspace: string\n\tresolveLocal?: (path: string) => LocalScriptContent | undefined\n\ttempScriptRefs?: Record<string, string>\n\t// Extra run args for a specific node (e.g. the uploaded S3Object bound to a\n\t// `data_upload` cascade root). Merged over `_wmill_skip_asset_dispatch`; all\n\t// other nodes run with empty inputs as before.\n\targsFor?: (path: string) => Record<string, any> | undefined\n\tonLaunched?: (path: string, jobId: string) => void\n}): (path: string) => Promise<string> {","sourceCodeStart":38,"sourceCodeEnd":74,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/frontend/src/lib/components/assets/AssetGraph/cascadeRun.ts#L38-L74","documentation":"cascadeRun.ts polls a launched Windmill job until it reaches a terminal state (success/failure/completed). makeWaitJobTerminal throws this timeout error when the job has not become terminal within CASCADE_JOB_TIMEOUT_MS (expressed in minutes in the message), while tolerating transient polling errors.","triggerScenarios":"A run triggered by the asset-graph cascade launched a job whose id never reaches a completed state before the timeout window elapses — the job is queued for a long time, the worker pool is saturated, the job hangs (e.g. waiting on an external service), or the workspace/job was cancelled in a way that never marks completion.","commonSituations":"Long queue on a self-hosted instance with few workers; a script stuck on a slow API call without its own timeout; gateway/auth issues making the poll endpoint permanently return non-completed states (poll errors are swallowed as 'transient'); running against the wrong workspace so the completed job is never seen.","solutions":["Check the job ${jobId} in the Windmill UI runs view to see if it is queued, running, hung, or failed silently","Scale up workers / clear the queue so the job actually runs within the timeout window","Add an internal timeout or `timeout` setting to the script itself so a hung step fails terminally instead of running forever","Increase CASCADE_JOB_TIMEOUT_MS in cascadeRun.ts if your jobs legitimately take longer than the default","Verify the poll requests hit the right workspace and aren't erroring permanently (the catch treats all poll failures as transient)"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Pre-check job state before committing to the wait\nconst job = await JobService.getJob({ workspace, id: jobId })\nif (job.completed && !job.success) throw new Error(`job ${jobId} already failed: ${job.logs ?? ''}`)","typeGuard":"function isTerminal(job: { completed?: boolean }): job is { completed: true } {\n  return job.completed === true\n}","tryCatchPattern":"try {\n  await waitJobTerminal(jobId)\n} catch (e) {\n  if (e instanceof Error && e.message.startsWith('Timed out after')) {\n    // inspect the job in the UI, cancel or retry with a longer window\n    await JobService.cancelJob({ workspace, id: jobId }).catch(() => {})\n    throw new Error(`cascade job ${jobId} hung; cancelled`, { cause: e })\n  }\n  throw e\n}","preventionTips":["Set per-step timeouts inside long-running scripts so they always reach a terminal state","Keep enough workers provisioned so cascade jobs don't sit queued past the timeout","Monitor the poll endpoint — repeated transient failures mean the job state will never be observed","Raise CASCADE_JOB_TIMEOUT_MS for pipelines with legitimately long jobs"],"tags":["timeout","jobs","polling","windmill"],"backgroundTag":"job-timeout","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}