apache/beam · error · IOException

Stream has been finished. Can not add any more elements.

Error message

Stream has been finished. Can not add any more elements.

What it means

BufferedElementCountingOutputStream counts elements written and emits length markers; once finished() is called the stream is closed for further output. markElementStart() checks the finished flag and throws IOException if a new element is started after finish. The stream is a one-shot sink: finishing is terminal.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java:134

    if (finished) {
      return;
    }
    flush();
    // Finish the stream with the terminatorValue.
    VarInt.encode(terminatorValue, os);
    if (!BUFFER_POOL.offer(buffer)) {
      // The pool is full, we can't store the buffer. We just drop the buffer.
    }
    finished = true;
  }

  /**
   * Marks that a new element is being output. This allows this output stream to use the buffer if
   * it had previously overflowed marking the start of a new block of elements.
   */
  public void markElementStart() throws IOException {
    if (finished) {
      throw new IOException("Stream has been finished. Can not add any more elements.");
    }
    count++;
  }

  @Override
  public void write(int b) throws IOException {
    if (finished) {
      throw new IOException("Stream has been finished. Can not write any more data.");
    }
    if (count == 0) {
      os.write(b);
      return;
    }

    if (buffer.hasRemaining()) {
      buffer.put((byte) b);
    } else {
      outputBuffer();

View on GitHub (pinned to 12126d8942)

Solutions

Create a new BufferedElementCountingOutputStream for each batch instead of reusing a finished instance.
Ensure finish() is called only after all elements have been marked and written; move finish() to the true end of the encode loop.
If reuse is required, track a 'finished' state in your wrapper and reinitialize the underlying stream before further writes.

Example fix

// before
stream.markElementStart(); writeElement(stream);
stream.finish();
stream.markElementStart(); // IOException
// after
stream.markElementStart(); writeElement(stream);
stream.finish();
stream = new BufferedElementCountingOutputStream(os, 20);
stream.markElementStart(); writeElement(stream);

Defensive patterns

Strategy: validation

Validate before calling

if (!stream.finished /* or your own flag */) { stream.markElementStart(); }

Try / catch

try (BufferedElementCountingOutputStream s = ...) {
  // write all elements
} // finish via close; never write afterwards

Prevention

Never reuse a finished stream; create a new one per batch.
Call finish() only after the last element is fully written.
Wrap lifecycle in try-with-resources or a owner object that serializes writes.

When it happens

Trigger: Calling markElementStart() after finish() — e.g. a caller keeps encoding elements after closing, or reuses the same output-stream wrapper for a second batch without creating a new instance.

Common situations: Reuse of a stream across batches or retries after finish; encode() loops whose termination condition runs one iteration past finish; custom sinks built on this stream that flush-then-finish but still receive data.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6842bd9e3d38ba58. Report an issue: GitHub.