apache/beam · error · ValueError

Transform '%s' was configured with unknown fields: %s. Valid

Error message

Transform '%s' was configured with unknown fields: %s. Valid fields: %s

What it means

When converting a Python dict configuration into a schema Row for a cross-language transform, Beam validates that every key matches a field of the transform's declared schema. Unknown keys (which may be misspellings or from a changed schema) raise this ValueError listing the extra keys and the valid field names, prefixed with the transform's identifier.

Source

Thrown at sdks/python/apache_beam/transforms/external.py:267

      elif type_info == 'array_type':
        return [
            dict_to_row_recursive(field_type.array_type.element_type, value)
            for value in py_value
        ]
      elif type_info == 'map_type':
        return {
            key: dict_to_row_recursive(field_type.map_type.value_type, value)
            for key, value in py_value.items()
        }
      else:
        return py_value

    def dict_to_row(schema_proto, py_value):
      row_type = named_tuple_from_schema(schema_proto)
      if isinstance(py_value, dict):
        extra = set(py_value.keys()) - set(row_type._fields)
        if extra:
          raise ValueError(
              f"Transform '{self.identifier()}' was configured with unknown "
              f"fields: {extra}. Valid fields: {set(row_type._fields)}")
        return row_type(
            *[
                dict_to_row_recursive(
                    field.type, py_value.get(field.name, None))
                for field in schema_proto.fields
            ])
      else:
        return row_type(py_value)

    return external_transforms_pb2.SchemaTransformPayload(
        identifier=self._identifier,
        configuration_schema=self._schema_proto,
        configuration_row=RowCoder(self._schema_proto).encode(
            dict_to_row(self._schema_proto, self._kwargs)))

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove or rename the offending keys to match the valid fields listed in the error message.
2. Check the transform's documentation for current field names after upgrading Beam or the expansion service.
3. Validate config dicts against the schema (via named_tuple_from_schema) before calling build().

Example fix

# before
builder.build(jar_path='x.jar', csv_file='data.csv')  # 'csv_file' unknown
# after
builder.build(jar_path='x.jar', file_pattern='data.csv')  # matches schema field

from apache_beam.typehints.schemas import named_tuple_from_schema
row_type = named_tuple_from_schema(schema_proto)
extra = set(config.keys()) - set(row_type._fields)
if extra:
    raise ValueError('unknown fields: %r' % extra)

def matches_schema(config, schema_proto) -> bool:
    fields = set(named_tuple_from_schema(schema_proto)._fields)
    return set(config.keys()) <= fields

try:
    row = builder.build(**config)
except ValueError as e:
    if 'unknown fields' in str(e):
        config = {k: v for k, v in config.items() if k in known_fields}
        row = builder.build(**config)
    else:
        raise

When it happens

Trigger: Passing a nested dict to a SchemaTransform build() (or dict_to_row) containing keys not present in the transform's schema, e.g. after renaming a field in a newer provider version.

Common situations: Upgrading a Beam expansion service whose transform schema changed; typos in config keys; hand-edited YAML/JSON configs converted to kwargs.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Could not find coder for URN " + urn
• Unknown timing constant: " + timing
• Unknown PaneInfo encoding 0x" + encoding.toString(16)
• Schema with id {schema.id} has encoding_positions_set=True,
• Attempted to encode null for non-nullable field "{}".

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/04e9572f221a1a0e. Report an issue: GitHub.