{"record":{"id":"5a9a23be04017eb1","repo":"apache/beam","slug":"arrow-map-key-field-cannot-be-nullable","errorCode":null,"errorMessage":"Arrow map key field cannot be nullable","messagePattern":"Arrow map key field cannot be nullable","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/typehints/arrow_type_compatibility.py","lineNumber":234,"sourceCode":"      nullable=beam_fieldtype.nullable,\n      metadata=metadata,\n  )\n\n\nif PYARROW_VERSION < (6, 0):\n  # In pyarrow < 6.0.0 we cannot construct a MapType object from Field\n  # instances, pa.map_ will only accept DataType instances. This makes it\n  # impossible to propagate nullability.\n  #\n  # Note this was changed in:\n  # https://github.com/apache/arrow/commit/64bef2ad8d9cd2fea122cfa079f8ca3fea8cdf5d\n  #\n  # Here we define a custom arrow map conversion function to handle these cases\n  # and error as appropriate.\n\n  def _make_arrow_map(beam_map_type: schema_pb2.MapType):\n    if beam_map_type.key_type.nullable:\n      raise TypeError('Arrow map key field cannot be nullable')\n    elif beam_map_type.value_type.nullable:\n      raise TypeError(\n          \"pyarrow<6 does not support creating maps with nullable \"\n          \"values. Please use pyarrow>=6.0.0\")\n\n    return pa.map_(\n        _arrow_type_from_beam_fieldtype(beam_map_type.key_type),\n        _arrow_type_from_beam_fieldtype(beam_map_type.value_type))\n\n  def _arrow_map_to_beam_map(arrow_map_type):\n    return schema_pb2.MapType(\n        key_type=_beam_fieldtype_from_arrow_type(arrow_map_type.key_type),\n        value_type=_beam_fieldtype_from_arrow_type(arrow_map_type.item_type))\n\nelse:\n\n  def _make_arrow_map(beam_map_type: schema_pb2.MapType):\n    return pa.map_(","sourceCodeStart":216,"sourceCodeEnd":252,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/typehints/arrow_type_compatibility.py#L216-L252","documentation":"When converting a Beam MapType to an Arrow map (for pyarrow < 6), _make_arrow_map rejects maps whose key type is marked nullable. Arrow requires map keys to be non-nullable, so Beam raises TypeError instead of producing an invalid schema.","triggerScenarios":"Converting a Beam schema to arrow via _arrow_type_from_beam_fieldtype where a map field was declared with nullable=True on its key_type — e.g. a typed dict row field where the key FieldType has nullable set, or a schema built programmatically/loaded from protobuf with nullable keys.","commonSituations":"Building Beam schemas via RowTypeConstraint/convert_to_schema where map key nullability got set by default; deserializing a schema protobuf from another SDK with nullable map keys; pyarrow<6 environments.","solutions":["Set the map key type non-nullable: build the FieldType with nullable=False for the key","Upgrade pyarrow to >= 6.0.0, which supports the modern map construction path","Strip nullable on keys when loading the schema (rebuild MapType with non-nullable key)"],"exampleFix":"// before\nfield_type = schema_pb2.FieldType(map_type=schema_pb2.MapType(\n    key_type=schema_pb2.FieldType(atomic_type=STRING, nullable=True),\n    value_type=...))\n// after\nfield_type = schema_pb2.FieldType(map_type=schema_pb2.MapType(\n    key_type=schema_pb2.FieldType(atomic_type=STRING, nullable=False),\n    value_type=...))","handlingStrategy":"validation","validationCode":"for field in schema.fields:\n    if field.type.HasField('map_type') and field.type.map_type.key_type.nullable:\n        raise ValueError(f'map key of field {field.name!r} must be non-nullable')","typeGuard":"def map_key_non_nullable(ft: schema_pb2.FieldType) -> bool:\n    return not (ft.WhichOneof('type_info') == 'map_type' and ft.map_type.key_type.nullable)","tryCatchPattern":"try:\n    arrow_type = _arrow_type_from_beam_fieldtype(beam_type)\nexcept TypeError as e:\n    if 'key field cannot be nullable' in str(e):\n        beam_type.map_type.key_type.nullable = False\n        arrow_type = _arrow_type_from_beam_fieldtype(beam_type)\n    else:\n        raise","preventionTips":["Always construct map key FieldTypes with nullable=False","Check schemas deserialized from other SDKs for nullable map keys","Prefer pyarrow>=6 where map handling is standardized"],"tags":["python","apache-beam","pyarrow","schema"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}