{"record":{"id":"d8d5bc5ca68f6d48","repo":"mastra-ai/mastra","slug":"factory-kickoff-is-waiting-on-a-run-that-has-not-e","errorCode":null,"errorMessage":"Factory kickoff is waiting on a run that has not ended.","messagePattern":"Factory kickoff is waiting on a run that has not ended\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"mastracode/factory/src/rules/dispatcher.ts","lineNumber":982,"sourceCode":"                    dedupeKey,\n                  },\n                  { ifActive: { behavior: 'deliver' }, ifIdle: { behavior: 'wake' }, requestContext },\n                ),\n              true,\n            );\n          try {\n            let settled = await sendKickoff(`factory-kickoff:${record.kickoffKey}`);\n            if (settled?.action === 'deliver') {\n              // `deliver` only proves the signal was queued onto a run already\n              // in flight. If that run ends without draining its queue the\n              // kickoff is dropped silently. There is no per-notification\n              // \"processed\" signal, so wait for the in-flight run to end and\n              // redeliver into the idle session unconditionally — the\n              // generation-scoped dedupeKey defeats inbox dedupe and the\n              // kickoff key keeps a duplicate run bounded, while a dropped\n              // kickoff strands the card forever.\n              if (!(await waitForAgentEndOrTimeout(agentEnd, this.#skillCompletionObservationTimeoutMs))) {\n                throw new Error('Factory kickoff is waiting on a run that has not ended.');\n              }\n              armAgentEnd();\n              settled = await sendKickoff(`factory-kickoff:${record.kickoffKey}:retry:${record.attempts}`);\n              if (settled?.action !== 'wake') {\n                throw new Error('Factory kickoff was queued onto an ending run and never reached the agent.');\n              }\n            }\n            const observed = await waitForAgentEndOrTimeout(agentEnd, this.#skillCompletionObservationTimeoutMs);\n            if (!observed) {\n              throw new Error('Factory kickoff run terminal event was not observed before timeout.');\n            } else if (endReason === 'error') {\n              throw new Error('Factory kickoff run ended in error.');\n            } else if (endReason === 'aborted') {\n              // Retryable for the same reason as skill decisions: the dominant\n              // cause is the process going away underneath the run, not a\n              // deliberate stop, and a spurious retry is bounded by\n              // MAX_ATTEMPTS while a dead card costs a human a manual nudge.\n              throw new Error('Factory kickoff run was aborted before it finished.');","sourceCodeStart":964,"sourceCodeEnd":1000,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/mastracode/factory/src/rules/dispatcher.ts#L964-L1000","documentation":"The dispatcher attempted a kickoff against a run that was still in flight. It waited for the agent to end via waitForAgentEndOrTimeout using #skillCompletionObservationTimeoutMs; if the run did not produce a terminal event within that window, the library throws so the pending start fails and the record is retried with an incremented attempts count and a new retry-suffixed kickoff key.","triggerScenarios":"An agent run on the target session outlived skillCompletionObservationTimeoutMs (long-running generation, hung model call, streaming stalls) so the observed agentEnd promise never resolved before the timeout.","commonSituations":"Model provider latency or stalled streams exceeding the observation timeout; timeout configured too aggressively for slow models; a run stuck because the process is under heavy load; debugging/breakpoints holding the run open during development.","solutions":["Increase skillCompletionObservationTimeoutMs (or the equivalent config) to exceed realistic run durations for your models.","Investigate why the run hangs — check model provider health, stream liveness, and trace the agent run for stalls.","Let the retry machinery re-deliver the kickoff (retry key factory-kickoff:<key>:retry:<attempts>) after the run ends.","Reduce concurrent load on the session so runs complete within the observation window."],"exampleFix":"// before: timeout too small for long generations\nnew FactoryDecisionDispatcher({ skillCompletionObservationTimeoutMs: 5_000, ... });\n// after: allow realistic run completion time\nnew FactoryDecisionDispatcher({ skillCompletionObservationTimeoutMs: 120_000, ... });","handlingStrategy":"retry","validationCode":"const runActive = await isRunActive(sessionId); // check run state before kickoff\nif (runActive && expectedRunDurationMs > observationTimeoutMs) {\n  throw new Error('run likely to outlive kickoff observation window');\n}","typeGuard":null,"tryCatchPattern":"try {\n  await dispatcher.dispatch(record);\n} catch (e) {\n  if (String(e?.message).includes('waiting on a run that has not ended')) {\n    await scheduleRetry(record); // dispatcher already backs off via attempts\n  } else throw e;\n}","preventionTips":["Set skillCompletionObservationTimeoutMs above your slowest realistic agent runs","Alert on runs that exceed the observation timeout — they usually indicate hung model calls","Avoid breakpoints/manual pauses in runs during development with dispatch enabled","Reduce concurrency on sessions that receive kickoffs"],"tags":["factory","timeout","async","agent-run"],"backgroundTag":"agent-run-timeout","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}