apache/beam · error · ValueError

Schema with id {schema.id} has encoding_positions_set=True,

Error message

Schema with id {schema.id} has encoding_positions_set=True,
            but found fields without encoding_position set: {names_no_pos}

What it means

When a Schema has encoding_positions_set=True, SchemaCoder requires every field to carry an encoding_position so fields can be encoded in a deterministic order. If positions were flagged as set but at least one field has encoding_position=None, Beam raises ValueError naming the offending fields, since the position list would be invalid. The check also guards against duplicate positions.

Source

Thrown at sdks/python/apache_beam/coders/coder_impl.py:1945

class RowCoderImpl(StreamCoderImpl):
  """For internal use only; no backwards-compatibility guarantees."""
  def __init__(self, schema, components):
    self.schema = schema
    self.num_fields = len(self.schema.fields)
    self.field_names = [f.name for f in self.schema.fields]
    self.field_nullable = [field.type.nullable for field in self.schema.fields]
    self.constructor = named_tuple_from_schema(schema)
    self.encoding_positions = list(range(len(self.schema.fields)))
    if self.schema.encoding_positions_set:
      # should never be duplicate encoding positions.
      enc_posx = list(
          set(field.encoding_position for field in self.schema.fields))
      if len(enc_posx) != len(self.schema.fields):
        names_no_pos = ", ".join(
            [f.name for f in self.schema.fields if f.encoding_position is None])
        raise ValueError(
            f'''Schema with id {schema.id} has encoding_positions_set=True,
            but found fields without encoding_position set: {names_no_pos}''')
      self.encoding_positions = list(
          field.encoding_position for field in self.schema.fields)
    self.encoding_positions_argsort = list(np.argsort(self.encoding_positions))
    self.encoding_positions_are_trivial = self.encoding_positions == list(
        range(len(self.encoding_positions)))
    self.components = list(
        components[self.encoding_positions.index(i)].get_impl()
        for i in self.encoding_positions)
    self.has_nullable_fields = any(
        field.type.nullable for field in self.schema.fields)

  def encode_to_stream(self, value, out, nested):
    out.write_var_int64(self.num_fields)
    attrs = [getattr(value, name) for name in self.field_names]

    if self.has_nullable_fields:

View on GitHub (pinned to 12126d8942)

Solutions

1. Assign an encoding_position to every field (call the schema API that sets positions consistently, e.g., 0..n-1)
2. Set encoding_positions_set=False to let Beam fall back to declaration order if explicit positions aren't needed
3. Rebuild the schema from the current field list so positions and the set-flag stay consistent
4. Fix schema-evolution code to reassign positions whenever fields are added/removed

Example fix

// before
schema = Schema(fields=[Field('a', int64), Field('b', str)])
schema.encoding_positions_set = True  # but no field has encoding_position
SchemaCoder(schema, ...)
// after
for i, f in enumerate(schema.fields):
    f.encoding_position = i
schema.encoding_positions_set = True
SchemaCoder(schema, ...)

def schema_positions_ok(schema) -> bool:
    if not schema.encoding_positions_set:
        return True
    return all(f.encoding_position is not None for f in schema.fields) and \
        len({f.encoding_position for f in schema.fields}) == len(schema.fields)

def all_fields_have_positions(schema) -> bool:
    return all(getattr(f, 'encoding_position', None) is not None for f in schema.fields)

try:
    coder = SchemaCoder(schema, tuple(coders))
except ValueError as e:
    if 'encoding_position' in str(e):
        for i, f in enumerate(schema.fields):
            f.encoding_position = i
        coder = SchemaCoder(schema, tuple(coders))
    else:
        raise

When it happens

Trigger: Constructing SchemaCoder (or decoding rows) from a Schema whose fields were marked encoding_positions_set=True while some fields lack encoding_position — e.g., a schema built programmatically where set_encoding_positions was half-applied, or a schema deserialized from a proto that claims positions are set but has None on new/renamed fields.

Common situations: Programmatic schema construction where fields were added after calling the position-setting API; schema evolution (adding fields) without reassigning encoding positions; schemas round-tripped through proto serialization with inconsistent flags; Beam version differences in how positions are defaulted.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Encountered unexpected value for null indicator: '%s'
• Integer value '{value}' is out of the encodable range for Va
• Cannot estimate size for integer value '{value}'. Value is o
• Number of components does not match number of coders.
• Cannot read state-written iterable without state reader.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f772921a99300fab. Report an issue: GitHub.