apache/beam · error · ValueError

Attempted to encode null for non-nullable field "{}".

Error message

Attempted to encode null for non-nullable field "{}".

What it means

apache_beam raises this ValueError when a Row (schema-encoded) value contains None for a field that was declared non-nullable (nullable=False) in the schema. The row coder encodes fields sequentially and explicitly rejects nulls where the schema does not permit them, because downstream consumers would not be able to decode a valid value for that field.

Source

Thrown at sdks/python/apache_beam/coders/coder_impl.py:1990

        running = 0
        for i, attr in enumerate(attrs):
          if i and i % 8 == 0:
            out.write_byte(running)
            running = 0
          running |= (attr is None) << (i % 8)
        out.write_byte(running)
      else:
        out.write_byte(0)
    else:
      out.write_byte(0)

    for i in range(self.num_fields):
      if not self.encoding_positions_are_trivial:
        i = self.encoding_positions_argsort[i]
      attr = attrs[i]
      if attr is None:
        if not self.field_nullable[i]:
          raise ValueError(
              "Attempted to encode null for non-nullable field \"{}\".".format(
                  self.schema.fields[i].name))
        continue
      component_coder = self.components[i]  # for typing
      component_coder.encode_to_stream(attr, out, True)

  def _row_column_encoders(self, columns):
    return [
        RowColumnEncoder.create(
            self.schema.fields[i].type.atomic_type,
            self.components[i],
            columns[name]) for i, name in enumerate(self.field_names)
    ]

  def encode_batch_to_stream(self, columns: Dict[str, np.ndarray], out):
    attrs = self._row_column_encoders(columns)
    n = len(next(iter(columns.values())))
    if self.has_nullable_fields:

View on GitHub (pinned to 12126d8942)

Solutions

  1. Make the field nullable in the schema (nullable=True) for the affected field name shown in the message
  2. Coalesce None to a sentinel/default before encoding (e.g. `x if x is not None else default`)
  3. Filter out records with the missing field before the coding point

Example fix

// before
beam.Create([{'id': 1, 'score': None}]).with_schema('id': int, 'score': int)
// after
beam.Create([{'id': 1, 'score': None}]) with schema field 'score' nullable=True, or {'id': 1, 'score': 0}
Defensive patterns

Strategy: validation

Validate before calling

for f in schema.fields:
    if not f.nullable:
        assert all(getattr(row, f.name) is not None for row in rows), f"null in non-nullable field {f.name}"

Type guard

def is_row_safe(row, schema):
    return all(getattr(row, f.name, None) is not None or f.nullable for f in schema.fields)

Prevention

When it happens

Trigger: Encoding a beam Row via a SchemaAwareCoderImpl/_RowEncoderImpl when an attribute is None but the corresponding field in the schema was created with nullable=False; typically when passing dicts/rows to Create() or a schema-bearing PTransform with missing values.

Common situations: Building rows from data sources with missing values (CSV blanks, missing JSON keys, left-outer-join results) while the schema (e.g. inferred or explicitly given to beam.RowTypeConstraint / Create) marks fields non-nullable; version changes where a field became NOT NULL.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8926514e2e4c6317. Report an issue: GitHub.