apache/beam · error · ValueError

Encountered unexpected value for null indicator: '%s'

Error message

Encountered unexpected value for null indicator: '%s'

What it means

NullableCoderImpl.decode_from_stream reads a single byte that indicates whether the encoded value is null (0x00), present (0x01), or something else. Any byte value other than ENCODE_NULL or ENCODE_PRESENT means the byte stream is not in the format this coder produced, so decoding cannot continue. Beam raises ValueError to fail fast on corrupt or mismatched encoded data rather than silently mis-decoding.

Source

Thrown at sdks/python/apache_beam/coders/coder_impl.py:828

      value_coder  # type: CoderImpl
  ):
    self._value_coder = value_coder

  def encode_to_stream(self, value, out, nested):
    if value is None:
      out.write_byte(self.ENCODE_NULL)
    else:
      out.write_byte(self.ENCODE_PRESENT)
      self._value_coder.encode_to_stream(value, out, nested)

  def decode_from_stream(self, in_stream, nested):
    null_indicator = in_stream.read_byte()
    if null_indicator == self.ENCODE_NULL:
      return None
    elif null_indicator == self.ENCODE_PRESENT:
      return self._value_coder.decode_from_stream(in_stream, nested)
    else:
      raise ValueError(
          "Encountered unexpected value for null indicator: '%s'" %
          null_indicator)

  def estimate_size(self, unused_value, nested=False):
    return 1 + (
        self._value_coder.estimate_size(unused_value)
        if unused_value is not None else 0)


class BigEndianShortCoderImpl(StreamCoderImpl):
  """For internal use only; no backwards-compatibility guarantees."""
  def encode_to_stream(self, value, out, nested):
    # type: (int, create_OutputStream, bool) -> None
    out.write_bigendian_int16(value)

  def decode_from_stream(self, in_stream, nested):
    # type: (create_InputStream, bool) -> int
    return in_stream.read_bigendian_int16()

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the bytes were produced by the same NullableCoder (and inner value coder) you are decoding with
2. Re-encode the data with the current Beam coder instead of reusing old/corrupt serialized bytes
3. Dump the offending byte (hex) at the failure position and compare against expected 0x00/0x01 to find misalignment
4. Check for truncation or offset errors in code that manually splits/concatenates encoded byte strings

Example fix

// before
raw = blob[:8]  # hand-sliced, misaligned
decoder.decode(raw)
// after
from apache_beam.coders import NullableCoder, VarIntCoder
coder = NullableCoder(VarIntCoder())
encoded = coder.encode(value)  # round-trip through the same coder
decoded = coder.decode(encoded)

def is_valid_nullable_encoding(data: bytes) -> bool:
    return bool(data) and data[0] in (0x00, 0x01)

def looks_like_nullable_encoded(data: object) -> bool:
    return isinstance(data, (bytes, bytearray)) and len(data) >= 1 and data[0] in (0, 1)

try:
    value = coder.decode(encoded)
except ValueError as e:
    if 'null indicator' in str(e):
        log.error('Corrupt/mismatched encoding: %s', e)
        value = None  # or re-encode from source
    else:
        raise

When it happens

Trigger: Calling NullableCoder.decode on a byte string/stream that was encoded with a different coder, that was truncated or corrupted, or that was produced by a non-Beam serializer; also happens when decoding raw bytes where a null-indicator byte position was misaligned (e.g., manual concatenation of encoded values).

Common situations: Hand-crafted or persisted encoded bytes replayed after a Beam upgrade or coder change; decoding data written by a pipeline using a different coder; byte-level manipulation of encoded PCollection snapshots; mixing encodings between runners or test fixtures.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• Integer value '{value}' is out of the encodable range for Va
• Cannot estimate size for integer value '{value}'. Value is o
• Number of components does not match number of coders.
• Cannot read state-written iterable without state reader.
• Invalid PaneInfoEncoding: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/48def170c5936ef2. Report an issue: GitHub.