apache/beam · error · TypeError

Arrow map key field cannot be nullable

Error message

Arrow map key field cannot be nullable

What it means

When converting a Beam MapType to an Arrow map (for pyarrow < 6), _make_arrow_map rejects maps whose key type is marked nullable. Arrow requires map keys to be non-nullable, so Beam raises TypeError instead of producing an invalid schema.

Solutions

1. Set the map key type non-nullable: build the FieldType with nullable=False for the key
2. Upgrade pyarrow to >= 6.0.0, which supports the modern map construction path
3. Strip nullable on keys when loading the schema (rebuild MapType with non-nullable key)

Example fix

// before
field_type = schema_pb2.FieldType(map_type=schema_pb2.MapType(
    key_type=schema_pb2.FieldType(atomic_type=STRING, nullable=True),
    value_type=...))
// after
field_type = schema_pb2.FieldType(map_type=schema_pb2.MapType(
    key_type=schema_pb2.FieldType(atomic_type=STRING, nullable=False),
    value_type=...))

for field in schema.fields:
    if field.type.HasField('map_type') and field.type.map_type.key_type.nullable:
        raise ValueError(f'map key of field {field.name!r} must be non-nullable')

def map_key_non_nullable(ft: schema_pb2.FieldType) -> bool:
    return not (ft.WhichOneof('type_info') == 'map_type' and ft.map_type.key_type.nullable)

try:
    arrow_type = _arrow_type_from_beam_fieldtype(beam_type)
except TypeError as e:
    if 'key field cannot be nullable' in str(e):
        beam_type.map_type.key_type.nullable = False
        arrow_type = _arrow_type_from_beam_fieldtype(beam_type)
    else:
        raise

When it happens

Trigger: Converting a Beam schema to arrow via _arrow_type_from_beam_fieldtype where a map field was declared with nullable=True on its key_type — e.g. a typed dict row field where the key FieldType has nullable set, or a schema built programmatically/loaded from protobuf with nullable keys.

Common situations: Building Beam schemas via RowTypeConstraint/convert_to_schema where map key nullability got set by default; deserializing a schema protobuf from another SDK with nullable map keys; pyarrow<6 environments.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• A schema is required to write non-schema'd data.
• All dicts in batch must have the same keys. extra keys
• An explicit schema is required to write non-schema'd…
• Attempted to encode null for non-nullable field
• Attribute fields not found in schema fields

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5a9a23be04017eb1. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/typehints/arrow_type_compatibility.py:234

      nullable=beam_fieldtype.nullable,
      metadata=metadata,
  )


if PYARROW_VERSION < (6, 0):
  # In pyarrow < 6.0.0 we cannot construct a MapType object from Field
  # instances, pa.map_ will only accept DataType instances. This makes it
  # impossible to propagate nullability.
  #
  # Note this was changed in:
  # https://github.com/apache/arrow/commit/64bef2ad8d9cd2fea122cfa079f8ca3fea8cdf5d
  #
  # Here we define a custom arrow map conversion function to handle these cases
  # and error as appropriate.

  def _make_arrow_map(beam_map_type: schema_pb2.MapType):
    if beam_map_type.key_type.nullable:
      raise TypeError('Arrow map key field cannot be nullable')
    elif beam_map_type.value_type.nullable:
      raise TypeError(
          "pyarrow<6 does not support creating maps with nullable "
          "values. Please use pyarrow>=6.0.0")

    return pa.map_(
        _arrow_type_from_beam_fieldtype(beam_map_type.key_type),
        _arrow_type_from_beam_fieldtype(beam_map_type.value_type))

  def _arrow_map_to_beam_map(arrow_map_type):
    return schema_pb2.MapType(
        key_type=_beam_fieldtype_from_arrow_type(arrow_map_type.key_type),
        value_type=_beam_fieldtype_from_arrow_type(arrow_map_type.item_type))

else:

  def _make_arrow_map(beam_map_type: schema_pb2.MapType):
    return pa.map_(

View on GitHub (pinned to 12126d8942)