apache/beam · error · RuntimeError

Can only iterate once over PrefetchingSourceSetIterable inst

Error message

Can only iterate once over PrefetchingSourceSetIterable instance.

What it means

PrefetchingSourceSetIterable streams a side-input source into an element queue via a reader thread, and is designed to be consumed exactly once. Calling __iter__ again after the first full iteration has started would yield an empty or partially-consumed view, so the class raises RuntimeError to protect this single-pass invariant.

Source

Thrown at sdks/python/apache_beam/runners/worker/sideinputs.py:151

                  self.element_queue.put(value)
                else:
                  self.element_queue.put(_globally_windowed(value))
        except queue.Empty:
          return
    except Exception as e:  # pylint: disable=broad-except
      _LOGGER.error(
          'Encountered exception in PrefetchingSourceSetIterable '
          'reader thread: %s',
          traceback.format_exc())
      self.reader_exceptions.put(e)
      self.has_errored = True
    finally:
      self.element_queue.put(READER_THREAD_IS_DONE_SENTINEL)

  def __iter__(self):
    # pylint: disable=too-many-nested-blocks
    if self.already_iterated:
      raise RuntimeError(
          'Can only iterate once over PrefetchingSourceSetIterable instance.')
    self.already_iterated = True

    # The invariants during execution are:
    # 1) A worker thread always posts the sentinel as the last thing it does
    #    before exiting.
    # 2) We always wait for all sentinels and then join all threads.
    num_readers_finished = 0
    try:
      while True:
        try:
          with self.read_counter:
            element = self.element_queue.get()
          if element is READER_THREAD_IS_DONE_SENTINEL:
            num_readers_finished += 1
            if num_readers_finished == self.num_reader_threads:
              return
          else:

View on GitHub (pinned to 12126d8942)

Solutions

Materialize the iterable into a list (list(iterable)) before consuming if you need multiple passes
Call .read() or obtain a fresh side input view for each access instead of reusing the iterable
Refactor the DoFn so the side input is iterated exactly once, accumulating results in one pass

Example fix

# before
for x in side_input_iterable:
  process(x)
for x in side_input_iterable:  # RuntimeError
  audit(x)
# after
items = list(side_input_iterable)
for x in items:
  process(x)
for x in items:
  audit(x)

Defensive patterns

Strategy: validation

Validate before calling

# before consuming a side input multiple times
if isinstance(view, PrefetchingSourceSetIterable) and view.already_iterated:
    raise RuntimeError('side input view already consumed; materialize it instead')
view = list(view)  # single materialization, reusable passes

Type guard

def is_consumable(it) -> bool:
    return not getattr(it, 'already_iterated', False)

Try / catch

try:
    for x in side_input:
        process(x)
except RuntimeError as e:
    if 'Can only iterate once' in str(e):
        side_input = get_fresh_side_input_view()
    else:
        raise

Prevention

Materialize side-input iterables to list() if more than one pass is needed
Never cache and reuse PrefetchingSourceSetIterable across bundle phases
Access side inputs via the standard view interfaces (AsList/AsDict) rather than raw iterables

When it happens

Trigger: Calling iter()/for-loops twice on the same PrefetchingSourceSetIterable, or re-iterating it after it was already consumed (self.already_iterated is True). Typically happens when a DoFn's start_bundle/process accesses the side input view multiple times without re-materializing it, or Beam SDK code reuses a cached iterable.

Common situations: User DoFns that iterate a side-input iterable twice (e.g., once to count, once to process); custom side-input access patterns in the Fn API worker; retry/replay logic in worker code that re-reads the same view.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/12cff868b09272f6. Report an issue: GitHub.