What causes "flush on closed file" in apache/beam?

Check writer.closed before calling flush(): only flush when the file handle is still open.

apache/beam · error · ValueError

flush on closed file

Error message

flush on closed file

What it means

AvroRowWriter.flush() raises ValueError when the underlying file handle is already closed. The wrapper refuses to flush an Avro writer into a closed file, since flushing after close would fail or corrupt output. This guards against double-close or write-after-close misuse in BigQuery file sinks.

Solutions

Check writer.closed before calling flush(): only flush when the file handle is still open.
Do not call flush() after close(); rely on close() to perform the final flush.
If managing manually, restructure to a single owner of the writer lifecycle so flush/close happen exactly once.

Example fix

// before
writer.close()
writer.flush()  # ValueError: flush on closed file

// after
writer.flush()
writer.close()
# or guard:
if not writer.closed:
    writer.flush()

Defensive patterns

Strategy: type-guard

Validate before calling

if writer.closed:
    raise RuntimeError('cannot flush: writer already closed')
writer.flush()

Type guard

def can_flush(writer):
    return not writer.closed

Try / catch

try:
    writer.flush()
except ValueError as e:
    if 'flush on closed file' not in str(e):
        raise
    # writer already closed; nothing to do

Prevention

Give a single code path ownership of flush/close for each writer.
Never call flush() after close(); call flush() before close() if needed.
Use try/finally so close() runs exactly once.
Check the .closed property before any lifecycle operation.

When it happens

Trigger: Calling flush() explicitly after close() was already called on the AvroRowWriter; calling close() twice, where the second close() internally flushes via the first path; using the writer in a context where the file handle was closed externally before flush.

Common situations: Writing Avro files for a BigQuery load job and manually closing the writer then calling flush in a finally block; a sink pipeline that closes the file and a caller that also flushes on teardown; refactored code that lost track of writer lifecycle.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/36bd59d07c797276. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:1530

      raise ValueError("Output stream must be writable")

    self._file_handle = file_handle
    avro_schema = fastavro.parse_schema(
        get_avro_schema_from_table_schema(schema))
    self._avro_writer = fastavro.write.Writer(self._file_handle, avro_schema)

  def close(self):
    if not self._file_handle.closed:
      self.flush()
      self._file_handle.close()

  @property
  def closed(self):
    return self._file_handle.closed

  def flush(self):
    if self._file_handle.closed:
      raise ValueError("flush on closed file")

    self._avro_writer.flush()
    self._file_handle.flush()

  def read(self, size=-1):
    raise io.UnsupportedOperation("AvroRowWriter is not readable")

  def tell(self):
    # Flush the fastavro Writer to the underlying stream, otherwise there isn't
    # a reliable way to determine how many bytes have been written.
    self._avro_writer.flush()
    return self._file_handle.tell()

  def writable(self):
    return self._file_handle.writable()

  def write(self, row):
    try:

View on GitHub (pinned to 12126d8942)