{"record":{"id":"48def170c5936ef2","repo":"apache/beam","slug":"encountered-unexpected-value-for-null-indicator-s","errorCode":null,"errorMessage":"Encountered unexpected value for null indicator: '%s'","messagePattern":"Encountered unexpected value for null indicator: '(.+?)'","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/coders/coder_impl.py","lineNumber":828,"sourceCode":"      value_coder  # type: CoderImpl\n  ):\n    self._value_coder = value_coder\n\n  def encode_to_stream(self, value, out, nested):\n    if value is None:\n      out.write_byte(self.ENCODE_NULL)\n    else:\n      out.write_byte(self.ENCODE_PRESENT)\n      self._value_coder.encode_to_stream(value, out, nested)\n\n  def decode_from_stream(self, in_stream, nested):\n    null_indicator = in_stream.read_byte()\n    if null_indicator == self.ENCODE_NULL:\n      return None\n    elif null_indicator == self.ENCODE_PRESENT:\n      return self._value_coder.decode_from_stream(in_stream, nested)\n    else:\n      raise ValueError(\n          \"Encountered unexpected value for null indicator: '%s'\" %\n          null_indicator)\n\n  def estimate_size(self, unused_value, nested=False):\n    return 1 + (\n        self._value_coder.estimate_size(unused_value)\n        if unused_value is not None else 0)\n\n\nclass BigEndianShortCoderImpl(StreamCoderImpl):\n  \"\"\"For internal use only; no backwards-compatibility guarantees.\"\"\"\n  def encode_to_stream(self, value, out, nested):\n    # type: (int, create_OutputStream, bool) -> None\n    out.write_bigendian_int16(value)\n\n  def decode_from_stream(self, in_stream, nested):\n    # type: (create_InputStream, bool) -> int\n    return in_stream.read_bigendian_int16()","sourceCodeStart":810,"sourceCodeEnd":846,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/coders/coder_impl.py#L810-L846","documentation":"NullableCoderImpl.decode_from_stream reads a single byte that indicates whether the encoded value is null (0x00), present (0x01), or something else. Any byte value other than ENCODE_NULL or ENCODE_PRESENT means the byte stream is not in the format this coder produced, so decoding cannot continue. Beam raises ValueError to fail fast on corrupt or mismatched encoded data rather than silently mis-decoding.","triggerScenarios":"Calling NullableCoder.decode on a byte string/stream that was encoded with a different coder, that was truncated or corrupted, or that was produced by a non-Beam serializer; also happens when decoding raw bytes where a null-indicator byte position was misaligned (e.g., manual concatenation of encoded values).","commonSituations":"Hand-crafted or persisted encoded bytes replayed after a Beam upgrade or coder change; decoding data written by a pipeline using a different coder; byte-level manipulation of encoded PCollection snapshots; mixing encodings between runners or test fixtures.","solutions":["Verify the bytes were produced by the same NullableCoder (and inner value coder) you are decoding with","Re-encode the data with the current Beam coder instead of reusing old/corrupt serialized bytes","Dump the offending byte (hex) at the failure position and compare against expected 0x00/0x01 to find misalignment","Check for truncation or offset errors in code that manually splits/concatenates encoded byte strings"],"exampleFix":"// before\nraw = blob[:8]  # hand-sliced, misaligned\ndecoder.decode(raw)\n// after\nfrom apache_beam.coders import NullableCoder, VarIntCoder\ncoder = NullableCoder(VarIntCoder())\nencoded = coder.encode(value)  # round-trip through the same coder\ndecoded = coder.decode(encoded)","handlingStrategy":"validation","validationCode":"def is_valid_nullable_encoding(data: bytes) -> bool:\n    return bool(data) and data[0] in (0x00, 0x01)","typeGuard":"def looks_like_nullable_encoded(data: object) -> bool:\n    return isinstance(data, (bytes, bytearray)) and len(data) >= 1 and data[0] in (0, 1)","tryCatchPattern":"try:\n    value = coder.decode(encoded)\nexcept ValueError as e:\n    if 'null indicator' in str(e):\n        log.error('Corrupt/mismatched encoding: %s', e)\n        value = None  # or re-encode from source\n    else:\n        raise","preventionTips":["Always round-trip through the same coder instance/version that encoded the bytes","Never manually slice or concatenate coder output at arbitrary offsets","When persisting encoded data, store the coder type alongside it"],"tags":["python","apache-beam","coding","decoding","data-corruption"],"backgroundTag":"invalid-enum-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}