apache/beam · error · RuntimeError

Cannot access the output of an error handler until it has be

Error message

Cannot access the output of an error handler until it has been closed.

What it means

ErrorHandler.output() returns the transformed error PCollections produced by applying the error consumer, but only after the handler has been 'closed' (its consumer can no longer be attached to more transforms). Apache Beam throws this RuntimeError because accessing the output before close() would expose PCollections that are not yet wired into the pipeline.

Source

Thrown at sdks/python/apache_beam/transforms/error_handling.py:82

  def __exit__(self, *exec_info):
    if exec_info[0] is None:
      self.close()

  def close(self):
    """Indicates all error-producing operations have reported any errors.

    Invokes the provided error consuming PTransform on any provided error
    PCollections.
    """
    self._output = (
        tuple(self._error_pcolls) | transforms.Flatten() | self._consumer)
    self._closed = True

  def output(self):
    """Returns result of applying the error consumer to the error pcollections.
    """
    if not self._closed:
      raise RuntimeError(
          "Cannot access the output of an error handler "
          "until it has been closed.")
    return self._output

  def add_error_pcollection(self, pcoll):
    """Called by a class implementing error handling on the error records.
    """
    pcoll.pipeline._register_error_handler(self)
    self._error_pcolls.append(pcoll)

  def verify_closed(self):
    """Called at end of pipeline construction to ensure errors are not ignored.
    """
    if not self._closed:
      raise RuntimeError(
          "Unclosed error handler initialized at %s" % self._creation_traceback)

View on GitHub (pinned to 12126d8942)

Solutions

Move the handler.output() call inside (or after) the with error_handling(...) block so close() has run first.
Ensure every ErrorHandler created is used as a context manager; do not call output() manually right after construction.
If you need error records, write them to a sink or read them after pipeline completion instead of accessing output() early.

Example fix

# before
handler = ErrorHandlingConfig(...)
result = handler.output()  # RuntimeError
# after
with error_handling(...) as handler:
  transformed = pcoll.with_exception_handling(...)
  result = handler.output()  # valid: handler closed on exit

Defensive patterns

Strategy: try-catch

Validate before calling

if not getattr(handler, '_closed', False):
    raise RuntimeError('call handler.output() only after the with-block closes it')

Type guard

def output_ready(handler) -> bool:
    return bool(getattr(handler, '_closed', False))

Try / catch

try:
    out = handler.output()
except RuntimeError:
    # handler not yet closed; finish pipeline construction first
    out = None

Prevention

Always use error handlers as context managers (with ... as handler:).
Only read handler.output() inside or after the with-block.
Never hoist output() out of pipeline construction code.

When it happens

Trigger: Calling handler.output() in a pipeline-construction block before the with_exception_handling(...) context manager exits (which sets _closed=True).

Common situations: Developers grabbing the collected error records mid-pipeline-construction, or hoisting output() out of the with-block to inspect results before Beam finalizes the handler.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0a1f323980bac0d0. Report an issue: GitHub.