What causes "Cannot write to a finished stream" in pola-rs/polars?

Restructure so finish() runs exactly once after the last write, and return early from error paths once finished

pola-rs/polars · error · PolarsError

UnexpectedEof

UnexpectedEof

Error message

Cannot write to a finished stream

What it means

Returned by StreamWriter::write in polars-arrow's IPC stream writer when a record batch is written after StreamWriter::finish has already been called. finish() writes the zero-length continuation that terminates an Arrow IPC stream and sets finished = true; every later write is rejected with this error. Note the io::ErrorKind is UnexpectedEof, which is misleading - nothing ran out of input, the writer is simply closed for further writes.

Source

Thrown at crates/polars-arrow/src/io/ipc/write/stream.rs:93

            ipc_message: schema_to_bytes(
                schema,
                self.ipc_fields.as_ref().unwrap(),
                self.custom_schema_metadata.as_deref(),
            ),
            arrow_data: vec![],
        };
        write_message(&mut self.writer, &encoded_message)?;
        Ok(())
    }

    /// Writes [`RecordBatchT`] to the stream
    pub fn write(
        &mut self,
        columns: &RecordBatchT<Box<dyn Array>>,
        ipc_fields: Option<&[IpcField]>,
    ) -> PolarsResult<()> {
        if self.finished {
            let io_err = std::io::Error::new(
                std::io::ErrorKind::UnexpectedEof,
                "Cannot write to a finished stream".to_string(),
            );
            return Err(PolarsError::from(io_err));
        }

        // we can't make it a closure because it borrows (and it can't borrow mut and non-mut below)
        #[allow(clippy::or_fun_call)]
        let fields = ipc_fields.unwrap_or(self.ipc_fields.as_ref().unwrap());

        let (encoded_dictionaries, encoded_message) = encode_chunk(
            columns,
            fields,
            &mut self.dictionary_tracker,
            &self.write_options,
        )?;

        for encoded_dictionary in encoded_dictionaries {

View on GitHub (pinned to df599052da)

Solutions

Restructure so finish() runs exactly once after the last write, and return early from error paths once finished
If more batches can still arrive, buffer them and write them to a fresh StreamWriter on a new stream/file - an IPC stream cannot be reopened
Track a finished flag next to the writer and skip or queue writes once it is set
If you wrap StreamWriter, expose an is_finished() accessor so callers can check before writing

Example fix

// before
for batch in batches {
    if let Err(e) = writer.write(&batch, None) {
        writer.finish()?;
    }
    // still writing after finish -> 'Cannot write to a finished stream'
}

// after
let mut finished = false;
for batch in batches {
    if !finished {
        writer.write(&batch, None)?;
    }
}
writer.finish()?;

Defensive patterns

Strategy: validation

Validate before calling

// Keep finish-state next to the writer and check before every write
let mut finished = false;
for batch in batches {
    if finished {
        continue; // or buffer / error explicitly
    }
    writer.write(&batch, None)?;
}
writer.finish()?;
finished = true;

Try / catch

match result {
    Err(PolarsError::IO(e)) if e.kind() == std::io::ErrorKind::UnexpectedEof
        && e.to_string().contains("finished stream") => break, // stop writing loop
    other => other?,
}

Prevention

Call finish() exactly once, at the outermost scope that owns the writer
Return early from error paths once the stream is finished
Never share one StreamWriter across retry attempts - create a new one per stream
Wrap StreamWriter in your own type that tracks and enforces finished state

When it happens

Trigger: Calling StreamWriter::write (or any higher-level batch-writing loop) after finish() was already invoked - e.g. an error path that finishes the stream and then falls through to write remaining batches, or a background producer still writing while the main thread closed the stream.

Common situations: Retry loops that reuse one writer across attempts; early-exit code that calls finish() inside the loop body instead of after it; concurrent producer/consumer designs where one side closes the stream; refactors that moved finish() into a helper invoked per-chunk.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/1cae18d922ef6cb1. Report an issue: GitHub.