apache/beam · error · RuntimeError

Bundle processing associated with

Error message

Bundle processing associated with %s has failed. Check prior failing response and attached exception for details.

What it means

BundleProcessorCache.lookup raises this when the requested instruction_id is recorded in failed_instruction_ids. The prior failure's exception is attached via 'from', so the original error is available as __cause__. The cache deliberately turns previously failed bundle lookups into loud errors rather than silent restarts.

Solutions

  1. Inspect the chained cause (__cause__) of this error — it holds the original bundle failure; fix that root problem.
  2. Check worker logs immediately before this error for the first failing response/exception of the bundle.
  3. Fix the failing DoFn/data issue so the bundle does not fail in the first place.
  4. Ensure the runner does not re-send control requests for a failed instruction id.

Example fix

# before
try:
    process_bundle(instruction_id)
except RuntimeError as e:
    print(e)  # only the wrapper message

# after
try:
    process_bundle(instruction_id)
except RuntimeError as e:
    print(e)
    if e.__cause__:
        raise e.__cause__  # surface the original bundle failure
Defensive patterns

Strategy: try-catch

Try / catch

try:
    processor = cache.lookup(instruction_id)
except RuntimeError as e:
    root = e.__cause__
    log.error('bundle %s failed earlier', instruction_id)
    if root:
        raise root  # fix the original failure, not the wrapper
    raise

Prevention

When it happens

Trigger: Any call to lookup(instruction_id) (e.g. from process_bundle or monitoring handlers) after the bundle previously threw and was discarded via discard(instruction_id, exception).

Common situations: A bundle fails mid-processing (user code exception, resource issue) and a subsequent control request (e.g. progress or finalize) references the same already-failed instruction; duplicate/late control messages after a failure.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/73c68e5284bf094a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/worker/sdk_worker.py:542

      except KeyError:
        # The instruction may have not been pre-registered before execution
        # since activate() may have never been invoked
        pass
    return processor

  def lookup(self, instruction_id):
    # type: (str) -> Optional[bundle_processor.BundleProcessor]

    """
    Return the requested ``BundleProcessor`` from the cache.

    Will return ``None`` if the BundleProcessor is known but not yet ready. Will
    raise an error if the ``instruction_id`` is not known or has been discarded.
    """
    with self._lock:
      if instruction_id in self.failed_instruction_ids:
        e = self.failed_instruction_ids[instruction_id]
        raise RuntimeError(
            'Bundle processing associated with %s has failed. '
            'Check prior failing response and attached exception for details.' %
            instruction_id) from e
      processor = self.active_bundle_processors.get(
          instruction_id, (None, None))[-1]
      if processor:
        return processor
      if instruction_id in self.known_not_running_instruction_ids:
        return None
      raise RuntimeError('Unknown process bundle id %s.' % instruction_id)

  def discard(self, instruction_id, exception):
    # type: (str, Exception) -> None

    """
    Marks the instruction id as failed shutting down the ``BundleProcessor``.
    """
    processor = None

View on GitHub (pinned to 12126d8942)