openai/codex · error · std::io::Error

InvalidData

InvalidData

Error message

remote MCP server output stream lost process events: expected sequence {expected_seq}, received {received_seq}

What it means

The transport consumes MCP-server stdout through a broadcast event stream; when the subscriber lags, recover_lagged_events() tries to replay retained executor output using last_seq as a read cursor. If the executor's retained-output log has already evicted the needed chunks, there is an unrecoverable gap and the transport closes itself (clearing buffers, setting closed) with InvalidData rather than splice truncated bytes into the JSON-RPC stream. The message names the first expected sequence number and the first number actually received after the gap.

Source

Thrown at codex-rs/rmcp-client/src/executor_process_transport.rs:392

        }
        self.last_seq = self.last_seq.max(response.next_seq.saturating_sub(1));
        if let Some(message) = response.failure {
            warn!(
                "Remote MCP server process failed ({}): {message}",
                self.program_name
            );
            self.closed = true;
        } else if response.closed {
            self.closed = true;
        }
        Ok(())
    }

    fn close_for_lost_output(&mut self, expected_seq: u64, received_seq: u64) -> io::Error {
        self.stdout.clear();
        self.stderr.clear();
        self.closed = true;
        io::Error::new(
            io::ErrorKind::InvalidData,
            format!(
                "remote MCP server output stream lost process events: expected sequence {expected_seq}, received {received_seq}"
            ),
        )
    }

    fn push_process_output_if_new(&mut self, chunk: ProcessOutputChunk) {
        if !self.should_accept_seq(chunk.seq) {
            return;
        }
        self.push_process_output(chunk);
    }

    fn push_process_output(&mut self, chunk: ProcessOutputChunk) {
        let bytes = chunk.chunk.into_inner();
        match chunk.stream {
            // MCP stdio uses stdout as the protocol stream. PTY output is

View on GitHub (pinned to 339751715c)

Solutions

Treat the connection as lost: tear down and restart the MCP server session (buffers are cleared and the transport is closed by design)
Reduce the server's stdout volume (log to stderr or a file instead — stderr is out-of-band)
Make sure the receive side is polled promptly; avoid long blocking work between polls
If you operate the executor, increase the retained-output/broadcast capacity so replay can cover lag

Example fix

// before
// keep polling a transport that logged 'lost process events'

// after
match transport.receive().await {
    Some(message) => handle(message),
    None => {
        // transport closed itself after an unrecoverable sequence gap
        warn!("MCP stdout gap; restarting server");
        restart_mcp_server().await?;
    }
}

Defensive patterns

Strategy: fallback

Type guard

fn is_lost_process_events(error: &std::io::Error) -> bool {
    error.kind() == std::io::ErrorKind::InvalidData
        && error.to_string().contains("lost process events")
}

Try / catch

// The transport self-closes on this error; recovery = rebuild the session
if let Err(error) = transport.recover_lagged_events().await {
    warn!("unrecoverable MCP output gap: {error}");
    drop(transport);
    let transport = reconnect_mcp_server(&config).await?;
}

Prevention

Keep the receive loop hot; never block between polls of the transport
Point chatty MCP servers at stderr or file logging, not stdout — stderr is out-of-band
If you operate the executor, size the retained-output buffer above worst-case stdout bursts
Monitor for 'output stream lagged' warnings — they precede the unrecoverable gap

When it happens

Trigger: receive_message() hits broadcast RecvError::Lagged and calls process.read(Some(last_seq), ...); recovery fails because a returned chunk has seq > last_seq + 1, or response.next_seq implies output chunks that were evicted. Happens when the broadcast channel overflows while the orchestrator event loop is stalled or the server floods stdout faster than retention holds it.

Common situations: A chatty MCP server emitting very large or frequent stdout bursts; a blocked or slow orchestrator event loop (slow downstream JSON parsing, debugger pauses); executor output-retention buffer sized too small for the server's output rate.

Related errors

AI-assisted analysis of openai/codex@339751715c (2026-08-25). Data as JSON: /api/errors/6aaf6ae75d82a9ab. Report an issue: GitHub.