{"record":{"id":"6aaf6ae75d82a9ab","repo":"openai/codex","slug":"invaliddata-6aaf6a","errorCode":"InvalidData","errorMessage":"remote MCP server output stream lost process events: expected sequence {expected_seq}, received {received_seq}","messagePattern":"remote MCP server output stream lost process events: expected sequence (.+?), received (.+?)","errorType":"exception","errorClass":"std::io::Error","httpStatus":null,"severity":"error","filePath":"codex-rs/rmcp-client/src/executor_process_transport.rs","lineNumber":392,"sourceCode":"        }\n        self.last_seq = self.last_seq.max(response.next_seq.saturating_sub(1));\n        if let Some(message) = response.failure {\n            warn!(\n                \"Remote MCP server process failed ({}): {message}\",\n                self.program_name\n            );\n            self.closed = true;\n        } else if response.closed {\n            self.closed = true;\n        }\n        Ok(())\n    }\n\n    fn close_for_lost_output(&mut self, expected_seq: u64, received_seq: u64) -> io::Error {\n        self.stdout.clear();\n        self.stderr.clear();\n        self.closed = true;\n        io::Error::new(\n            io::ErrorKind::InvalidData,\n            format!(\n                \"remote MCP server output stream lost process events: expected sequence {expected_seq}, received {received_seq}\"\n            ),\n        )\n    }\n\n    fn push_process_output_if_new(&mut self, chunk: ProcessOutputChunk) {\n        if !self.should_accept_seq(chunk.seq) {\n            return;\n        }\n        self.push_process_output(chunk);\n    }\n\n    fn push_process_output(&mut self, chunk: ProcessOutputChunk) {\n        let bytes = chunk.chunk.into_inner();\n        match chunk.stream {\n            // MCP stdio uses stdout as the protocol stream. PTY output is","sourceCodeStart":374,"sourceCodeEnd":410,"githubUrl":"https://github.com/openai/codex/blob/339751715c64496cb86246bfb3935f40e309dd3d/codex-rs/rmcp-client/src/executor_process_transport.rs#L374-L410","documentation":"The transport consumes MCP-server stdout through a broadcast event stream; when the subscriber lags, recover_lagged_events() tries to replay retained executor output using last_seq as a read cursor. If the executor's retained-output log has already evicted the needed chunks, there is an unrecoverable gap and the transport closes itself (clearing buffers, setting closed) with InvalidData rather than splice truncated bytes into the JSON-RPC stream. The message names the first expected sequence number and the first number actually received after the gap.","triggerScenarios":"receive_message() hits broadcast RecvError::Lagged and calls process.read(Some(last_seq), ...); recovery fails because a returned chunk has seq > last_seq + 1, or response.next_seq implies output chunks that were evicted. Happens when the broadcast channel overflows while the orchestrator event loop is stalled or the server floods stdout faster than retention holds it.","commonSituations":"A chatty MCP server emitting very large or frequent stdout bursts; a blocked or slow orchestrator event loop (slow downstream JSON parsing, debugger pauses); executor output-retention buffer sized too small for the server's output rate.","solutions":["Treat the connection as lost: tear down and restart the MCP server session (buffers are cleared and the transport is closed by design)","Reduce the server's stdout volume (log to stderr or a file instead — stderr is out-of-band)","Make sure the receive side is polled promptly; avoid long blocking work between polls","If you operate the executor, increase the retained-output/broadcast capacity so replay can cover lag"],"exampleFix":"// before\n// keep polling a transport that logged 'lost process events'\n\n// after\nmatch transport.receive().await {\n    Some(message) => handle(message),\n    None => {\n        // transport closed itself after an unrecoverable sequence gap\n        warn!(\"MCP stdout gap; restarting server\");\n        restart_mcp_server().await?;\n    }\n}","handlingStrategy":"fallback","validationCode":null,"typeGuard":"fn is_lost_process_events(error: &std::io::Error) -> bool {\n    error.kind() == std::io::ErrorKind::InvalidData\n        && error.to_string().contains(\"lost process events\")\n}","tryCatchPattern":"// The transport self-closes on this error; recovery = rebuild the session\nif let Err(error) = transport.recover_lagged_events().await {\n    warn!(\"unrecoverable MCP output gap: {error}\");\n    drop(transport);\n    let transport = reconnect_mcp_server(&config).await?;\n}","preventionTips":["Keep the receive loop hot; never block between polls of the transport","Point chatty MCP servers at stderr or file logging, not stdout — stderr is out-of-band","If you operate the executor, size the retained-output buffer above worst-case stdout bursts","Monitor for 'output stream lagged' warnings — they precede the unrecoverable gap"],"tags":["rust","mcp","stdio-transport","sequence-gap","broadcast-channel","data-loss"],"backgroundTag":"broadcast-channel-lagged","analyzedSha":"339751715c64496cb86246bfb3935f40e309dd3d","analyzedAt":"2026-08-25T05:35:09.876Z","schemaVersion":2},"datasetVersion":"2026-08-25T06:17:31.827Z"}