{"record":{"id":"d7cf7cdf32c9cda2","repo":"mastra-ai/mastra","slug":"factory-kickoff-was-queued-onto-an-ending-run-and","errorCode":null,"errorMessage":"Factory kickoff was queued onto an ending run and never reached the agent.","messagePattern":"Factory kickoff was queued onto an ending run and never reached the agent\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"mastracode/factory/src/rules/dispatcher.ts","lineNumber":987,"sourceCode":"            );\n          try {\n            let settled = await sendKickoff(`factory-kickoff:${record.kickoffKey}`);\n            if (settled?.action === 'deliver') {\n              // `deliver` only proves the signal was queued onto a run already\n              // in flight. If that run ends without draining its queue the\n              // kickoff is dropped silently. There is no per-notification\n              // \"processed\" signal, so wait for the in-flight run to end and\n              // redeliver into the idle session unconditionally — the\n              // generation-scoped dedupeKey defeats inbox dedupe and the\n              // kickoff key keeps a duplicate run bounded, while a dropped\n              // kickoff strands the card forever.\n              if (!(await waitForAgentEndOrTimeout(agentEnd, this.#skillCompletionObservationTimeoutMs))) {\n                throw new Error('Factory kickoff is waiting on a run that has not ended.');\n              }\n              armAgentEnd();\n              settled = await sendKickoff(`factory-kickoff:${record.kickoffKey}:retry:${record.attempts}`);\n              if (settled?.action !== 'wake') {\n                throw new Error('Factory kickoff was queued onto an ending run and never reached the agent.');\n              }\n            }\n            const observed = await waitForAgentEndOrTimeout(agentEnd, this.#skillCompletionObservationTimeoutMs);\n            if (!observed) {\n              throw new Error('Factory kickoff run terminal event was not observed before timeout.');\n            } else if (endReason === 'error') {\n              throw new Error('Factory kickoff run ended in error.');\n            } else if (endReason === 'aborted') {\n              // Retryable for the same reason as skill decisions: the dominant\n              // cause is the process going away underneath the run, not a\n              // deliberate stop, and a spurious retry is bounded by\n              // MAX_ATTEMPTS while a dead card costs a human a manual nudge.\n              throw new Error('Factory kickoff run was aborted before it finished.');\n            }\n          } finally {\n            unsubscribe();\n          }\n        },","sourceCodeStart":969,"sourceCodeEnd":1005,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/mastracode/factory/src/rules/dispatcher.ts#L969-L1005","documentation":"During a kick-off retry against an in-flight run, the dispatcher queued the kickoff message onto the run as it was ending. sendKickoff returns a settled result whose action must be 'wake' to prove the message reached the agent; any other action (or no result) means the kickoff was swallowed by a terminating run and the card would strand. The library throws so the record is retried with a fresh dedupe-safe kickoff key.","triggerScenarios":"sendKickoff resolved with settled.action !== 'wake' — the run terminated between armAgentEnd() and delivery, the inbox rejected the message via dedupe, or the delivery landed after the agent stopped accepting input.","commonSituations":"Race between run teardown and message delivery under load; duplicate kickoff messages where generation-scoped dedupeKey suppressed a retry delivery; process crash mid-delivery; attempting wake on a run that already emitted its end event.","solutions":["Rely on the built-in retry: the throw marks the pending start failed with a retryAt schedule; ensure MAX_ATTEMPTS is not exhausted.","Verify the inbox/dedupe layer is not swallowing retried kickoffs (each retry uses ':retry:<attempts>' in the key — check the key is unique).","Check agent/session logs for the run's end event timing; if teardown routinely races delivery, serialize kickoff after confirmed run end.","Ensure the agent process is stable (no crashes mid-run) — process restarts are the dominant cause per the surrounding code."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  await dispatcher.dispatch(record);\n} catch (e) {\n  if (String(e?.message).includes('never reached the agent')) {\n    await scheduleRetry(record); // bounded by MAX_ATTEMPTS; kickoff key is attempt-suffixed\n  } else throw e;\n}","preventionTips":["Keep MAX_ATTEMPTS and retry backoff configured so transient delivery races self-heal","Don't bypass the generation-scoped dedupeKey mechanism with hand-rolled keys","Ensure the agent process drains gracefully so runs don't terminate mid-delivery","Watch for inbox dedupe suppressing retried kickoffs in logs"],"tags":["factory","async","race-condition","kickoff"],"backgroundTag":"message-lost-during-run-teardown","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}