apache/beam · critical · ValueError

Source did not produce expected values when performing a ree

Error message

Source did not produce expected values when performing a reentrant read after reading %d values. Expected %r received %r.

What it means

Raised by assert_reentrant_reads_succeed when, after pausing the original read at element i, starting a second read, and then continuing the original iterator, the completed original read does not match the expected full set of values. This means the source's read iterator is disturbed by the interleaved reentrant read, i.e. the source is not safe for reentrant (concurrent/interleaved) reads — typically shared mutable iterator/reader state.

Source

Thrown at sdks/python/apache_beam/io/source_test_utils.py:232

  for i in range(1, len(expected_values) - 1):
    read_iter = source.read(
        source.get_range_tracker(start_position, stop_position))
    original_read = []
    for _ in range(i):
      original_read.append(next(read_iter))

    # Reentrant read
    reentrant_read = [
        val for val in source.read(
            source.get_range_tracker(start_position, stop_position))
    ]

    # Continuing original read.
    for val in read_iter:
      original_read.append(val)

    if equal_to(original_read)(expected_values):
      raise ValueError(
          'Source did not produce expected values when '
          'performing a reentrant read after reading %d values. '
          'Expected %r received %r.' % (i, expected_values, original_read))

    if equal_to(reentrant_read)(expected_values):
      raise ValueError(
          'A reentrant read of source after reading %d values '
          'did not produce expected values. Expected %r '
          'received %r.' % (i, expected_values, reentrant_read))


def assert_split_at_fraction_behavior(
    source, num_items_to_read_before_split, split_fraction, expected_outcome):
  """Verifies the behaviour of splitting a source at a given fraction.

  Asserts that splitting a :class:`~apache_beam.io.iobase.BoundedSource` either
  fails after reading **num_items_to_read_before_split** items, or succeeds in
  a way that is consistent according to

View on GitHub (pinned to 12126d8942)

Solutions

1. Make read() fully independent: each call must open its own reader/file handle and track its own position.
2. Move mutable state (file offsets, cursors) from shared/instance attributes into the per-read iterator closure.
3. Ensure the RangeTracker passed to read() is the only source of position state and is not shared across reads.
4. Add a test reading two iterators from the same source interleaved to confirm isolation.

Example fix

# before: shared cursor state
def read(self, range_tracker):
  for r in self._records[self._cursor:]:
    self._cursor += 1
    yield r
# after: derive from range_tracker per read
def read(self, range_tracker):
  start = range_tracker.start_position()
  stop = range_tracker.stop_position()
  for r in self._records[start:stop]:
    yield r

it1 = src.read(rt1); it2 = src.read(rt2)
assert list(it1) == list(it2), "interleaved read isolation broken"

def is_stateless_reader(source):
    return not any(isinstance(getattr(source, a, None), (io.IOBase,)) for a in dir(source))

try:
    source_test_utils.assert_reentrant_reads_succeed((src, None, None))
except ValueError as e:
    logger.error("reentrant read broke original iterator: %s", e)
    raise

When it happens

Trigger: assert_reentrant_reads_succeed reads i elements, performs a full second source.read(), then exhausts the first iterator; equal_to(original_read)(expected_values) detects the original read yielded wrong/missing/duplicated values.

Common situations: A custom BoundedSource whose read() returns a generator that mutates shared state (file handle offset, class-level cursor) so a second read() call corrupts the first read's progress.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Source is too trivial since it produces only %d values. Plea
• A reentrant read of source after reading %d values did not p
• BigQuery source must be split before being read
• Reference source must produce the same number of records as
• Reference source and provided list of sources must produce t

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/644eeb24816a126f. Report an issue: GitHub.