apache/beam · error · RuntimeError

Unknown process bundle id

Error message

Unknown process bundle id %s.

What it means

BundleProcessorCache.lookup returns None for known-but-not-running instructions, but if the instruction_id is in neither failed_instruction_ids, active_bundle_processors, nor known_not_running_instruction_ids, it raises RuntimeError 'Unknown process bundle id'. The cache has never seen this instruction, indicating a control-protocol or lifecycle violation.

Solutions

  1. Align runner and SDK harness Beam versions to keep control-protocol state machines in sync.
  2. Check worker logs for a harness restart/crash that would have wiped the in-memory cache.
  3. Verify the runner does not issue control requests (finalize, split, progress) for completed or never-started bundles.
  4. Retry the work item on the runner side; if reproducible, capture the instruction id and file a Beam issue.
Defensive patterns

Strategy: type-guard

Validate before calling

# confirm the instruction was actually started before touching the cache
assert instruction_id in started_instruction_ids, \
    f'{instruction_id} was never registered with this harness'

Type guard

def is_known_instruction(cache, instruction_id: str) -> bool:
    with cache._lock:
        return (instruction_id in cache.failed_instruction_ids
                or instruction_id in cache.active_bundle_processors
                or instruction_id in cache.known_not_running_instruction_ids)

Try / catch

try:
    proc = cache.lookup(instruction_id)
except RuntimeError as e:
    if 'Unknown process bundle id' in str(e):
        requeue_work(instruction_id)  # harness lost state; re-execute
    raise

Prevention

When it happens

Trigger: lookup() called with an instruction id that was never registered (no process_bundle started for it) or whose records were fully evicted after completion/finalization.

Common situations: Runner and SDK harness version skew causing mismatched instruction bookkeeping; duplicate or delayed control messages referencing long-finished bundles; a runner bug sending finalize/progress for wrong ids; harness restart losing cache state.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a2f7922ae199ce39. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/worker/sdk_worker.py:552

    Return the requested ``BundleProcessor`` from the cache.

    Will return ``None`` if the BundleProcessor is known but not yet ready. Will
    raise an error if the ``instruction_id`` is not known or has been discarded.
    """
    with self._lock:
      if instruction_id in self.failed_instruction_ids:
        e = self.failed_instruction_ids[instruction_id]
        raise RuntimeError(
            'Bundle processing associated with %s has failed. '
            'Check prior failing response and attached exception for details.' %
            instruction_id) from e
      processor = self.active_bundle_processors.get(
          instruction_id, (None, None))[-1]
      if processor:
        return processor
      if instruction_id in self.known_not_running_instruction_ids:
        return None
      raise RuntimeError('Unknown process bundle id %s.' % instruction_id)

  def discard(self, instruction_id, exception):
    # type: (str, Exception) -> None

    """
    Marks the instruction id as failed shutting down the ``BundleProcessor``.
    """
    processor = None
    with self._lock:
      tb_str = "".join(traceback.format_exception(exception))
      if len(tb_str) > 10240:
        tb_str = (
            tb_str[:5000] + "\n... [traceback truncated] ...\n" +
            tb_str[-5000:])
      clean_exception = RuntimeError(
          f"Original Exception: {type(exception).__name__}: {str(exception)[:2000]}\n{tb_str}"
      )
      self.failed_instruction_ids[instruction_id] = clean_exception

View on GitHub (pinned to 12126d8942)