{"record":{"id":"6a201b46e66e206a","repo":"apache/beam","slug":"unrecognized-arrow-type-arrow-type-r","errorCode":null,"errorMessage":"Unrecognized arrow type: {arrow_type!r}","messagePattern":"Unrecognized arrow type: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/typehints/arrow_type_compatibility.py","lineNumber":136,"sourceCode":"  elif isinstance(arrow_type, pa.ListType):\n    return schema_pb2.FieldType(\n        array_type=schema_pb2.ArrayType(\n            element_type=_beam_fieldtype_from_arrow_field(\n                arrow_type.value_field)))\n  elif isinstance(arrow_type, pa.MapType):\n    return schema_pb2.FieldType(map_type=_arrow_map_to_beam_map(arrow_type))\n  elif isinstance(arrow_type, pa.StructType):\n    return schema_pb2.FieldType(\n        row_type=schema_pb2.RowType(\n            schema=schema_pb2.Schema(\n                fields=[\n                    _beam_field_from_arrow_field(arrow_type[i])\n                    for i in range(len(arrow_type))\n                ],\n            )))\n\n  else:\n    raise ValueError(f\"Unrecognized arrow type: {arrow_type!r}\")\n\n\ndef _option_as_arrow_metadata(beam_option: schema_pb2.Option, *,\n                              prefix: bytes) -> Tuple[bytes, bytes]:\n  return (\n      prefix + beam_option.name.encode('UTF-8'),\n      beam_option.SerializeToString())\n\n\n_field_option_as_arrow_metadata = partial(\n    _option_as_arrow_metadata, prefix=BEAM_FIELD_OPTION_KEY_PREFIX)\n_schema_option_as_arrow_metadata = partial(\n    _option_as_arrow_metadata, prefix=BEAM_SCHEMA_OPTION_KEY_PREFIX)\n\n\ndef arrow_schema_from_beam_schema(beam_schema: schema_pb2.Schema) -> pa.Schema:\n  return pa.schema(\n      [_arrow_field_from_beam_field(field) for field in beam_schema.fields],","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/typehints/arrow_type_compatibility.py#L118-L154","documentation":"_beam_fieldtype_from_arrow_type converts a pyarrow type to a Beam schema FieldType but only handles a subset of arrow types (primitives, lists, structs, maps). Any other arrow type (e.g. dictionary, union, decimal on old pyarrow) falls to the else branch and raises ValueError with the repr of the arrow type.","triggerScenarios":"Passing a pa.Table whose schema contains an arrow type Beam cannot map — e.g. pa.dictionary, pa.union, extension types, large_string on old pyarrow — into code paths like arrow batch conversion or schema inference that call _beam_fieldtype_from_arrow_field/_arrow_map_to_beam_map.","commonSituations":"Reading a Parquet file with dictionary-encoded columns into a Table then converting to Beam schema; pyarrow version differences introducing types (large_binary, extension types); pandas-to-arrow inference producing unsupported types.","solutions":["Cast unsupported columns before conversion, e.g. table.cast to primitive types or .dictionary_decode() dictionary columns","Pin/use a pyarrow version where the type maps to Beam (and upgrade Beam for newer arrow type support)","Inspect the offending arrow type in the message and replace it in your data source (e.g. write plain strings instead of dictionary types)"],"exampleFix":"// before\ntable = pq.read_table(path)  # contains dictionary-encoded column\n// after\ntable = pq.read_table(path)\ntable = table.set_column(\n    table.schema.get_field_index('col'),\n    'col', table.column('col').cast(pa.string()))","handlingStrategy":"type-guard","validationCode":"SUPPORTED = {pa.int8(), pa.int16(), pa.int32(), pa.int64(), pa.uint8(), pa.uint16(), pa.uint32(), pa.uint64(), pa.float32(), pa.float64(), pa.string(), pa.binary(), pa.bool_(), pa.timestamp('us'), pa.date32()}\nbad = [f.name for f in table.schema if f.type not in SUPPORTED and not (pa.types.is_struct(f.type) or pa.types.is_list(f.type) or pa.types.is_map(f.type))]\nif bad:\n    raise TypeError(f'unsupported arrow columns: {bad}')","typeGuard":"def is_convertible(t: pa.DataType) -> bool:\n    return (pa.types.is_struct(t) or pa.types.is_list(t) or pa.types.is_map(t) or\n            pa.types.is_dictionary(t) is False and t in SUPPORTED)","tryCatchPattern":"try:\n    beam_type = _beam_fieldtype_from_arrow_type(arrow_type)\nexcept ValueError:\n    arrow_type = arrow_type.dictionary_decode() if pa.types.is_dictionary(arrow_type) else arrow_type.cast(pa.string())","preventionTips":["Dictionary-decode or cast dictionary/extension columns right after reading Parquet","Pin pyarrow versions tested with your Beam release","Inspect table.schema before converting to Beam schemas"],"tags":["python","apache-beam","pyarrow","type-conversion"],"backgroundTag":"incompatible-source-type","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}