{"record":{"id":"6f8c68ff4f122a0a","repo":"windmill-labs/windmill","slug":"timed-out-after-math-round-cascade-job-timeout-m-6f8c68","errorCode":null,"errorMessage":"Timed out after ${Math.round(CASCADE_JOB_TIMEOUT_MS / 60000)}min waiting for job ${jobId} to finish","messagePattern":"Timed out after (.+?)min waiting for job (.+?) to finish","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte","lineNumber":1576,"sourceCode":"\t// `runDraftAwareCascade`'s finally clears the running-root guard.\n\tconst CASCADE_POLL_INTERVAL_MS = 1000\n\tconst CASCADE_JOB_TIMEOUT_MS = 30 * 60 * 1000\n\tasync function waitJobTerminal(jobId: string): Promise<'success' | 'failure'> {\n\t\tconst deadline = Date.now() + CASCADE_JOB_TIMEOUT_MS\n\t\twhile (Date.now() < deadline) {\n\t\t\ttry {\n\t\t\t\tconst r = await JobService.getCompletedJobResultMaybe({\n\t\t\t\t\tworkspace: $workspaceStore!,\n\t\t\t\t\tid: jobId,\n\t\t\t\t\tgetStarted: false\n\t\t\t\t})\n\t\t\t\tif (r.completed) return r.success ? 'success' : 'failure'\n\t\t\t} catch {\n\t\t\t\t// transient — retry on the next tick\n\t\t\t}\n\t\t\tawait new Promise((res) => setTimeout(res, CASCADE_POLL_INTERVAL_MS))\n\t\t}\n\t\tthrow new Error(\n\t\t\t`Timed out after ${Math.round(CASCADE_JOB_TIMEOUT_MS / 60000)}min waiting for job ${jobId} to finish`\n\t\t)\n\t}\n\t// \"Run + downstream\" over a chain that includes drafts: the backend\n\t// asset-trigger dispatcher only resolves deployed rows, so the page\n\t// orchestrates the closure itself (topological order over the resolved\n\t// graph the user is looking at). Deployed-only chains never come here —\n\t// they keep the production dispatcher (see onRunProducer).\n\tasync function runDraftAwareCascade(rootPath: string): Promise<string | undefined> {\n\t\tif (cascadeRunningRoot) {\n\t\t\tsendUserToast(`A chain run from ${cascadeRunningRoot} is still in progress`, true)\n\t\t\treturn undefined\n\t\t}\n\t\tconst closure = computeDownstreamClosure(graphWithDraft, rootPath)\n\t\tif (closure.cyclic.length > 0) {\n\t\t\tsendUserToast(\n\t\t\t\t`Not running ${closure.cyclic.length} script(s) on a dependency cycle: ${closure.cyclic.join(', ')}`,\n\t\t\t\ttrue","sourceCodeStart":1558,"sourceCodeEnd":1594,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte#L1558-L1594","documentation":"The pipeline page polls a cascade job's status every CASCADE_POLL_INTERVAL_MS and gives up after CASCADE_JOB_TIMEOUT_MS minutes. When the deadline elapses without the job reporting completed, the page throws this timeout error so the caller knows the orchestration (run + downstream chain) did not finish, not that the job necessarily failed.","triggerScenarios":"Calling the page's run/downstream cascade orchestration whose job (e.g. asset-trigger dispatcher closure over deployed rows + drafts) takes longer than CASCADE_JOB_TIMEOUT_MS (minutes) to complete, or whose status endpoint keeps throwing transient errors so the poll never observes completion.","commonSituations":"Very large asset graphs or slow downstream runs exceeding the fixed timeout; backend worker congestion delaying the job; transient network errors hitting the poll endpoint on every tick; a stale jobId whose job was cancelled and will never report completed.","solutions":["Increase CASCADE_JOB_TIMEOUT_MS in the pipeline page if your cascades legitimately take longer","Check the backend job by id for failure/cancellation — the job may be dead rather than slow","Verify the backend instance and workers are healthy (job stuck in queue)","Re-run the cascade; transient poll failures reset nothing but the job may finish on a second attempt"],"exampleFix":"// before\nthrow new Error(`Timed out after ${Math.round(CASCADE_JOB_TIMEOUT_MS / 60000)}min waiting for job ${jobId} to finish`)\n// after\n// raise the ceiling for large graphs\nconst CASCADE_JOB_TIMEOUT_MS = 30 * 60 * 1000 // was 5 * 60 * 1000","handlingStrategy":"retry","validationCode":"const started = Date.now()\nconst remainingMs = CASCADE_JOB_TIMEOUT_MS - (Date.now() - started)\nif (remainingMs <= 0) throw new Error(`Cascade job ${jobId} timed out`)\n// also check the job's real status before waiting:\nconst r = await fetchJob(jobId)\nif (r.completed) return r.success ? 'success' : 'failure'","typeGuard":null,"tryCatchPattern":"try {\n  await waitForCascadeJob(jobId)\n} catch (e) {\n  if (String(e.message).includes('Timed out after')) {\n    const status = await fetchJobStatus(jobId) // inspect real state, maybe still running\n    sendUserToast(`Cascade still pending: job ${jobId} (${status})`, true)\n  } else throw e\n}","preventionTips":["Size CASCADE_JOB_TIMEOUT_MS to your largest expected cascade, not the average","Monitor job queue depth so slow workers are caught before timeouts","Alert on jobs stuck past a fraction of the timeout rather than only at the deadline","Surface jobId in UI toasts so users can inspect the stuck job"],"tags":["timeout","polling","pipeline","frontend"],"backgroundTag":"job-poll-timeout","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}