apache/beam · error · ValueError

Unrecognized arrow type

Error message

Unrecognized arrow type: {arrow_type!r}

What it means

_beam_fieldtype_from_arrow_type converts a pyarrow type to a Beam schema FieldType but only handles a subset of arrow types (primitives, lists, structs, maps). Any other arrow type (e.g. dictionary, union, decimal on old pyarrow) falls to the else branch and raises ValueError with the repr of the arrow type.

Solutions

  1. Cast unsupported columns before conversion, e.g. table.cast to primitive types or .dictionary_decode() dictionary columns
  2. Pin/use a pyarrow version where the type maps to Beam (and upgrade Beam for newer arrow type support)
  3. Inspect the offending arrow type in the message and replace it in your data source (e.g. write plain strings instead of dictionary types)

Example fix

// before
table = pq.read_table(path)  # contains dictionary-encoded column
// after
table = pq.read_table(path)
table = table.set_column(
    table.schema.get_field_index('col'),
    'col', table.column('col').cast(pa.string()))
Defensive patterns

Strategy: type-guard

Validate before calling

SUPPORTED = {pa.int8(), pa.int16(), pa.int32(), pa.int64(), pa.uint8(), pa.uint16(), pa.uint32(), pa.uint64(), pa.float32(), pa.float64(), pa.string(), pa.binary(), pa.bool_(), pa.timestamp('us'), pa.date32()}
bad = [f.name for f in table.schema if f.type not in SUPPORTED and not (pa.types.is_struct(f.type) or pa.types.is_list(f.type) or pa.types.is_map(f.type))]
if bad:
    raise TypeError(f'unsupported arrow columns: {bad}')

Type guard

def is_convertible(t: pa.DataType) -> bool:
    return (pa.types.is_struct(t) or pa.types.is_list(t) or pa.types.is_map(t) or
            pa.types.is_dictionary(t) is False and t in SUPPORTED)

Try / catch

try:
    beam_type = _beam_fieldtype_from_arrow_type(arrow_type)
except ValueError:
    arrow_type = arrow_type.dictionary_decode() if pa.types.is_dictionary(arrow_type) else arrow_type.cast(pa.string())

Prevention

When it happens

Trigger: Passing a pa.Table whose schema contains an arrow type Beam cannot map — e.g. pa.dictionary, pa.union, extension types, large_string on old pyarrow — into code paths like arrow batch conversion or schema inference that call _beam_fieldtype_from_arrow_field/_arrow_map_to_beam_map.

Common situations: Reading a Parquet file with dictionary-encoded columns into a Table then converting to Beam schema; pyarrow version differences introducing types (large_binary, extension types); pandas-to-arrow inference producing unsupported types.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6a201b46e66e206a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/typehints/arrow_type_compatibility.py:136

  elif isinstance(arrow_type, pa.ListType):
    return schema_pb2.FieldType(
        array_type=schema_pb2.ArrayType(
            element_type=_beam_fieldtype_from_arrow_field(
                arrow_type.value_field)))
  elif isinstance(arrow_type, pa.MapType):
    return schema_pb2.FieldType(map_type=_arrow_map_to_beam_map(arrow_type))
  elif isinstance(arrow_type, pa.StructType):
    return schema_pb2.FieldType(
        row_type=schema_pb2.RowType(
            schema=schema_pb2.Schema(
                fields=[
                    _beam_field_from_arrow_field(arrow_type[i])
                    for i in range(len(arrow_type))
                ],
            )))

  else:
    raise ValueError(f"Unrecognized arrow type: {arrow_type!r}")


def _option_as_arrow_metadata(beam_option: schema_pb2.Option, *,
                              prefix: bytes) -> Tuple[bytes, bytes]:
  return (
      prefix + beam_option.name.encode('UTF-8'),
      beam_option.SerializeToString())


_field_option_as_arrow_metadata = partial(
    _option_as_arrow_metadata, prefix=BEAM_FIELD_OPTION_KEY_PREFIX)
_schema_option_as_arrow_metadata = partial(
    _option_as_arrow_metadata, prefix=BEAM_SCHEMA_OPTION_KEY_PREFIX)


def arrow_schema_from_beam_schema(beam_schema: schema_pb2.Schema) -> pa.Schema:
  return pa.schema(
      [_arrow_field_from_beam_field(field) for field in beam_schema.fields],

View on GitHub (pinned to 12126d8942)