{"record":{"id":"dace26fd5517e9cc","repo":"mastra-ai/mastra","slug":"factory-kickoff-run-terminal-event-was-not-observe","errorCode":null,"errorMessage":"Factory kickoff run terminal event was not observed before timeout.","messagePattern":"Factory kickoff run terminal event was not observed before timeout\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"mastracode/factory/src/rules/dispatcher.ts","lineNumber":992,"sourceCode":"              // in flight. If that run ends without draining its queue the\n              // kickoff is dropped silently. There is no per-notification\n              // \"processed\" signal, so wait for the in-flight run to end and\n              // redeliver into the idle session unconditionally — the\n              // generation-scoped dedupeKey defeats inbox dedupe and the\n              // kickoff key keeps a duplicate run bounded, while a dropped\n              // kickoff strands the card forever.\n              if (!(await waitForAgentEndOrTimeout(agentEnd, this.#skillCompletionObservationTimeoutMs))) {\n                throw new Error('Factory kickoff is waiting on a run that has not ended.');\n              }\n              armAgentEnd();\n              settled = await sendKickoff(`factory-kickoff:${record.kickoffKey}:retry:${record.attempts}`);\n              if (settled?.action !== 'wake') {\n                throw new Error('Factory kickoff was queued onto an ending run and never reached the agent.');\n              }\n            }\n            const observed = await waitForAgentEndOrTimeout(agentEnd, this.#skillCompletionObservationTimeoutMs);\n            if (!observed) {\n              throw new Error('Factory kickoff run terminal event was not observed before timeout.');\n            } else if (endReason === 'error') {\n              throw new Error('Factory kickoff run ended in error.');\n            } else if (endReason === 'aborted') {\n              // Retryable for the same reason as skill decisions: the dominant\n              // cause is the process going away underneath the run, not a\n              // deliberate stop, and a spurious retry is bounded by\n              // MAX_ATTEMPTS while a dead card costs a human a manual nudge.\n              throw new Error('Factory kickoff run was aborted before it finished.');\n            }\n          } finally {\n            unsubscribe();\n          }\n        },\n      );\n      const completed = await this.#storage.completePendingStart(leaseIdentity(record, this.#ownerId), new Date());\n      if (!completed) throw new Error('Factory kickoff lease was lost before completion.');\n    } catch (error) {\n      await this.#storage.failPendingStart({","sourceCodeStart":974,"sourceCodeEnd":1010,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/mastracode/factory/src/rules/dispatcher.ts#L974-L1010","documentation":"After (re)delivering the kickoff, the dispatcher subscribes to the run's end and waits up to skillCompletionObservationTimeoutMs for the terminal event. If the agent end signal never arrives in that window, the library cannot confirm the run finished, so it throws to fail the pending start and schedule a retry. Without this guard a lost end event (e.g. dropped event after a process restart) would hang the dispatcher forever.","triggerScenarios":"The run's end event was never observed: the agent process died before emitting a terminal event, the event subscription missed the signal (unsubscribe race), or the run genuinely exceeded the observation timeout.","commonSituations":"Server restart or crash under an active Factory run; event bus/connection interruption losing the terminal event; timeouts set below the run's actual duration; a hung agent that never ends.","solutions":["Confirm the agent process stayed alive for the whole run; restart-driven loss should be picked up by the retry path.","Raise skillCompletionObservationTimeoutMs to cover your slowest realistic runs.","Check that the end-event subscription is established before the run starts (armAgentEnd ordering) so the terminal event cannot be missed.","Inspect event-bus/transport logs for dropped terminal events."],"exampleFix":"// before: subscribe after starting the run (event may be missed)\nconst run = await startAgent();\narmAgentEnd();\n// after: arm the end signal first\narmAgentEnd();\nconst run = await startAgent();","handlingStrategy":"retry","validationCode":"const observationMs = dispatcherOptions.skillCompletionObservationTimeoutMs;\nif (observationMs < longestExpectedRunMs) {\n  throw new Error(`observation timeout ${observationMs}ms too low; expected runs up to ${longestExpectedRunMs}ms`);\n}","typeGuard":null,"tryCatchPattern":"try {\n  await dispatcher.dispatch(record);\n} catch (e) {\n  if (String(e?.message).includes('terminal event was not observed')) {\n    await scheduleRetry(record);\n  } else throw e;\n}","preventionTips":["Arm the agent-end subscription before starting the run so terminal events can't be missed","Size the observation timeout to your slowest model runs","Ensure process restarts are detected and pending starts are failed promptly","Monitor event transport for dropped end events"],"tags":["factory","timeout","events","agent-run"],"backgroundTag":"terminal-event-not-observed","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}