{"record":{"id":"4d0bfdbd477db99f","repo":"mastra-ai/mastra","slug":"stdout-backpressure-exceeded-the-experiment-deadli","errorCode":null,"errorMessage":"stdout backpressure exceeded the experiment deadline","messagePattern":"stdout backpressure exceeded the experiment deadline","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/cli/src/commands/experiment/runtime.ts","lineNumber":385,"sourceCode":"    };\n  };\n  const abortForProtocolFailure = (message: string) => {\n    if (protocolFailure || terminal) return;\n    protocolFailure = new Error(message);\n    report(message);\n    controller?.abort(protocolFailure);\n  };\n  const waitForDrain = async () => {\n    const drainController = new AbortController();\n    let deadlineExceeded = false;\n    const clearTimer = scheduleAtDeadline(() => {\n      deadlineExceeded = true;\n      drainController.abort();\n    });\n    try {\n      await once(stdout, 'drain', { signal: drainController.signal });\n    } catch (error) {\n      if (deadlineExceeded) throw new Error('stdout backpressure exceeded the experiment deadline');\n      throw error;\n    } finally {\n      clearTimer();\n      drainController.abort();\n    }\n  };\n  const writeEvent = (type: string, payload: Record<string, unknown>) => {\n    if (!correlation || terminal || (finishing && type !== 'terminal') || (type === 'heartbeat' && heartbeatQueued)) {\n      return writeTail;\n    }\n    const event = {\n      ...correlation,\n      eventId: createEventId(),\n      sequence,\n      emittedAt: now().toISOString(),\n      type,\n      payload,\n    };","sourceCodeStart":367,"sourceCodeEnd":403,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/cli/src/commands/experiment/runtime.ts#L367-L403","documentation":"The experiment worker writes NDJSON protocol frames to stdout and, when the stream buffer is full (write() returned false), waits for 'drain' with an AbortController tied to the run deadline. If the deadline fires before stdout drains, the wait is aborted and this error is thrown, which is then wrapped into a ProtocolOutputError. It means the consumer of the worker's stdout stopped reading (or was too slow) for so long that the experiment's time budget expired mid-write.","triggerScenarios":"stdout.write() returns false (backpressure) and once(stdout,'drain') is aborted by the deadline timer before the pipe consumer drains the stream — e.g. the parent process is paused, blocked, or reading too slowly while the run deadline elapses.","commonSituations":"Parent CLI/runner blocked on something else and not consuming stdout; output redirected to a full or slow disk/file; downstream log collector stalled; a very chatty experiment emitting large frames faster than the consumer reads, combined with a tight deadline.","solutions":["Ensure the parent process continuously reads/consumes the worker's stdout (don't pause or block the pipe reader).","Increase the experiment deadline if runs legitimately produce large output volumes.","Reduce output volume (fewer/smaller events, larger heartbeat intervals) to lower backpressure.","Check where stdout is redirected — writing to a full disk or a stalled collector is a common cause."],"exampleFix":"// before (parent)\nchild.spawn(...); // stdout never read -> pipe fills\n// after\nchild.stdout.on('data', frame => handleFrame(frame)); // drain continuously","handlingStrategy":"try-catch","validationCode":"// before starting the worker, ensure a continuous stdout consumer exists\nchild.stdout.on('data', () => {}); // or wire a real frame parser; never leave the pipe unread","typeGuard":"null","tryCatchPattern":"try {\n  const frames = await runExperimentWorker({ mastra, runExperiment, build });\n} catch (e) {\n  if (e instanceof Error && e.message.includes('backpressure exceeded the experiment deadline')) {\n    // retry with a longer deadline or a lighter-output run\n  } else throw e;\n}","preventionTips":["Always consume the worker's stdout continuously from the parent process","Set a realistic experiment deadline relative to expected output volume","Redirect stdout to a fast, non-full destination","Reduce per-event payload size and event frequency"],"tags":["streaming","backpressure","timeout"],"backgroundTag":"stdout-backpressure","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}