apache/beam · error · TypeError

pyarrow<6 does not support creating maps with nullable…

Error message

pyarrow<6 does not support creating maps with nullable values. Please use pyarrow>=6.0.0

What it means

In pyarrow < 6, pa.map_ cannot create maps with nullable value fields, so Beam's _make_arrow_map raises TypeError telling the user to use pyarrow>=6.0.0 when converting a Beam map with nullable value type.

Solutions

  1. Upgrade pyarrow: pip install 'pyarrow>=6.0.0'
  2. Declare the map value type non-nullable (nullable=False) in the Beam schema
  3. Upgrade Beam itself, which may relax/adjust the conversion requirements

Example fix

// before
value_type=schema_pb2.FieldType(atomic_type=STRING, nullable=True)
// after
value_type=schema_pb2.FieldType(atomic_type=STRING, nullable=False)
# or: pip install 'pyarrow>=6.0.0'
Defensive patterns

Strategy: validation

Validate before calling

import pyarrow as pa
from packaging.version import parse
if parse(pa.__version__) < parse('6.0.0'):
    raise RuntimeError(f'pyarrow {pa.__version__} too old; need >=6.0.0 for nullable map values')

Type guard

def supports_nullable_map_values() -> bool:
    import pyarrow as pa
    return tuple(int(x) for x in pa.__version__.split('.')[:2]) >= (6, 0)

Try / catch

try:
    arrow_type = _arrow_type_from_beam_fieldtype(beam_map_type)
except TypeError as e:
    if 'pyarrow<6' in str(e):
        beam_map_type.value_type.nullable = False
        arrow_type = _arrow_type_from_beam_fieldtype(beam_map_type)
    else:
        raise

Prevention

When it happens

Trigger: Converting a Beam schema to arrow with pyarrow < 6 where a map field's value_type.nullable is True — the default when Beam marks value elements nullable (common for Optional/Any values).

Common situations: Old pyarrow environments (pre-6.0.0) in pinned Airflow/Beam deployments; Beam schemas derived from Optional dict values marked nullable; running with an outdated pyarrow installed alongside Beam.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4cbfbfb393be7a59. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/typehints/arrow_type_compatibility.py:236

  )


if PYARROW_VERSION < (6, 0):
  # In pyarrow < 6.0.0 we cannot construct a MapType object from Field
  # instances, pa.map_ will only accept DataType instances. This makes it
  # impossible to propagate nullability.
  #
  # Note this was changed in:
  # https://github.com/apache/arrow/commit/64bef2ad8d9cd2fea122cfa079f8ca3fea8cdf5d
  #
  # Here we define a custom arrow map conversion function to handle these cases
  # and error as appropriate.

  def _make_arrow_map(beam_map_type: schema_pb2.MapType):
    if beam_map_type.key_type.nullable:
      raise TypeError('Arrow map key field cannot be nullable')
    elif beam_map_type.value_type.nullable:
      raise TypeError(
          "pyarrow<6 does not support creating maps with nullable "
          "values. Please use pyarrow>=6.0.0")

    return pa.map_(
        _arrow_type_from_beam_fieldtype(beam_map_type.key_type),
        _arrow_type_from_beam_fieldtype(beam_map_type.value_type))

  def _arrow_map_to_beam_map(arrow_map_type):
    return schema_pb2.MapType(
        key_type=_beam_fieldtype_from_arrow_type(arrow_map_type.key_type),
        value_type=_beam_fieldtype_from_arrow_type(arrow_map_type.item_type))

else:

  def _make_arrow_map(beam_map_type: schema_pb2.MapType):
    return pa.map_(
        _arrow_field_from_beam_fieldtype(beam_map_type.key_type),
        _arrow_field_from_beam_fieldtype(beam_map_type.value_type))

View on GitHub (pinned to 12126d8942)