{"record":{"id":"4cbfbfb393be7a59","repo":"apache/beam","slug":"pyarrow-6-does-not-support-creating-maps-with-nullable","errorCode":null,"errorMessage":"pyarrow<6 does not support creating maps with nullable values. Please use pyarrow>=6.0.0","messagePattern":"pyarrow<6 does not support creating maps with nullable values\\. Please use pyarrow>=6\\.0\\.0","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/typehints/arrow_type_compatibility.py","lineNumber":236,"sourceCode":"  )\n\n\nif PYARROW_VERSION < (6, 0):\n  # In pyarrow < 6.0.0 we cannot construct a MapType object from Field\n  # instances, pa.map_ will only accept DataType instances. This makes it\n  # impossible to propagate nullability.\n  #\n  # Note this was changed in:\n  # https://github.com/apache/arrow/commit/64bef2ad8d9cd2fea122cfa079f8ca3fea8cdf5d\n  #\n  # Here we define a custom arrow map conversion function to handle these cases\n  # and error as appropriate.\n\n  def _make_arrow_map(beam_map_type: schema_pb2.MapType):\n    if beam_map_type.key_type.nullable:\n      raise TypeError('Arrow map key field cannot be nullable')\n    elif beam_map_type.value_type.nullable:\n      raise TypeError(\n          \"pyarrow<6 does not support creating maps with nullable \"\n          \"values. Please use pyarrow>=6.0.0\")\n\n    return pa.map_(\n        _arrow_type_from_beam_fieldtype(beam_map_type.key_type),\n        _arrow_type_from_beam_fieldtype(beam_map_type.value_type))\n\n  def _arrow_map_to_beam_map(arrow_map_type):\n    return schema_pb2.MapType(\n        key_type=_beam_fieldtype_from_arrow_type(arrow_map_type.key_type),\n        value_type=_beam_fieldtype_from_arrow_type(arrow_map_type.item_type))\n\nelse:\n\n  def _make_arrow_map(beam_map_type: schema_pb2.MapType):\n    return pa.map_(\n        _arrow_field_from_beam_fieldtype(beam_map_type.key_type),\n        _arrow_field_from_beam_fieldtype(beam_map_type.value_type))","sourceCodeStart":218,"sourceCodeEnd":254,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/typehints/arrow_type_compatibility.py#L218-L254","documentation":"In pyarrow < 6, pa.map_ cannot create maps with nullable value fields, so Beam's _make_arrow_map raises TypeError telling the user to use pyarrow>=6.0.0 when converting a Beam map with nullable value type.","triggerScenarios":"Converting a Beam schema to arrow with pyarrow < 6 where a map field's value_type.nullable is True — the default when Beam marks value elements nullable (common for Optional/Any values).","commonSituations":"Old pyarrow environments (pre-6.0.0) in pinned Airflow/Beam deployments; Beam schemas derived from Optional dict values marked nullable; running with an outdated pyarrow installed alongside Beam.","solutions":["Upgrade pyarrow: pip install 'pyarrow>=6.0.0'","Declare the map value type non-nullable (nullable=False) in the Beam schema","Upgrade Beam itself, which may relax/adjust the conversion requirements"],"exampleFix":"// before\nvalue_type=schema_pb2.FieldType(atomic_type=STRING, nullable=True)\n// after\nvalue_type=schema_pb2.FieldType(atomic_type=STRING, nullable=False)\n# or: pip install 'pyarrow>=6.0.0'","handlingStrategy":"validation","validationCode":"import pyarrow as pa\nfrom packaging.version import parse\nif parse(pa.__version__) < parse('6.0.0'):\n    raise RuntimeError(f'pyarrow {pa.__version__} too old; need >=6.0.0 for nullable map values')","typeGuard":"def supports_nullable_map_values() -> bool:\n    import pyarrow as pa\n    return tuple(int(x) for x in pa.__version__.split('.')[:2]) >= (6, 0)","tryCatchPattern":"try:\n    arrow_type = _arrow_type_from_beam_fieldtype(beam_map_type)\nexcept TypeError as e:\n    if 'pyarrow<6' in str(e):\n        beam_map_type.value_type.nullable = False\n        arrow_type = _arrow_type_from_beam_fieldtype(beam_map_type)\n    else:\n        raise","preventionTips":["Pin pyarrow>=6.0.0 in requirements","Declare map value types non-nullable when values are guaranteed present","Check pa.__version__ at pipeline startup"],"tags":["python","apache-beam","pyarrow","version-compatibility"],"backgroundTag":"missing-dependency","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}