apache/beam · error · ValueError

Cannot read state-written iterable without state reader.

Error message

Cannot read state-written iterable without state reader.

What it means

IterableCoderImpl.decode_from_stream supports a streaming encoding where a count of -1 means the remaining elements live in runner-managed state, accessed via a state reader supplied at construction (self._read_state). If a -1 marker appears in the byte stream but no state reader was configured, Beam raises ValueError because the state-backed portion of the iterable cannot be materialized.

Source

Thrown at sdks/python/apache_beam/coders/coder_impl.py:1369

    # type: (create_InputStream, bool) -> Sequence
    size = in_stream.read_bigendian_int32()

    if size >= 0:
      elements = [
          self._elem_coder.decode_from_stream(in_stream, True)
          for _ in range(size)
      ]  # type: Iterable[Any]
    else:
      elements = []
      count = in_stream.read_var_int64()
      while count > 0:
        for _ in range(count):
          elements.append(self._elem_coder.decode_from_stream(in_stream, True))
        count = in_stream.read_var_int64()

      if count == -1:
        if self._read_state is None:
          raise ValueError(
              'Cannot read state-written iterable without state reader.')

        state_token = in_stream.read_all(True)
        elements = _ConcatSequence(
            elements, self._read_state(state_token, self._elem_coder))

    return self._construct_from_sequence(elements)

  def estimate_size(self, value, nested=False):
    # type: (Any, bool) -> int

    """Estimates the encoded size of the given value, in bytes."""
    # TODO(ccy): This ignores element sizes.
    estimated_size, _ = (self.get_estimated_size_and_observables(value))
    return estimated_size

  def get_estimated_size_and_observables(self, value, nested=False):
    # type: (Any, bool) -> Tuple[int, Observables]

View on GitHub (pinned to 12126d8942)

Solutions

Decode such iterables in the context that provides a state reader (runner/Fn Harness state API), not with a standalone coder
Use the non-state (materialized) iterable encoding for offline decoding — fully materialize elements on the writer side
If constructing IterableCoderImpl manually, pass a read_state callable that can resolve state tokens
Check pipeline options / runner support so state-backed coders are only used where the state client exists

Example fix

// before
coder_impl = IterableCoderImpl(VarIntCoder().get_impl())  # no state reader
coder_impl.decode_from_stream(stream_with_state_marker, True)  # ValueError
// after
coder_impl = IterableCoderImpl(
    VarIntCoder().get_impl(),
    read_state=lambda token, elem_coder: fetch_state_elements(token, elem_coder))
coder_impl.decode_from_stream(stream_with_state_marker, True)

Defensive patterns

Strategy: try-catch

Validate before calling

def is_state_marked_encoded(stream) -> bool:
    # cannot peek portably; instead check construction: coder needs _read_state for state bytes
    return getattr(coder_impl, '_read_state', None) is not None

Type guard

def supports_state_read(coder_impl) -> bool:
    return getattr(coder_impl, '_read_state', None) is not None

Try / catch

try:
    values = coder_impl.decode_from_stream(stream, True)
except ValueError as e:
    if 'state reader' in str(e):
        log.error('Decode state-backed iterable in runner context, not standalone: %s', e)
        values = []
    else:
        raise

Prevention

Only use state-backed iterable coders inside a runner/Fn Harness that supplies a state reader
For offline decoding, use fully materialized (count-prefixed) iterables
When constructing coder impls manually in tests, always wire a read_state callable if state bytes are possible

When it happens

Trigger: Decoding an iterable encoded with the state-written (count == -1) format using an IterableCoderImpl created without a state reader — e.g., decoding state-protocol bytes with a plain coder, or a test harness/direct pipeline path lacking the state reader wiring.

Common situations: Replaying or unit-testing encoded iterables captured from a streaming runner that uses the state API; decoding Fn Harness state protocol payloads outside the runner; Beam version/config mismatch where state-backed iterables are produced but the local decode path has no _read_state.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/becc8cba7d83b189. Report an issue: GitHub.