{"record":{"id":"b5d5e3303678d2e2","repo":"apache/beam","slug":"element-type-element-type-must-be-compatible-with-beam","errorCode":null,"errorMessage":"Element type {element_type} must be compatible with Beam Schemas (https://beam.apache.org/documentation/programming-guide/#schemas) for batch type pa.Table.","messagePattern":"Element type (.+?) must be compatible with Beam Schemas \\(https://beam\\.apache\\.org/documentation/programming-guide/#schemas\\) for batch type pa\\.Table\\.","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/typehints/arrow_type_compatibility.py","lineNumber":314,"sourceCode":"\n\nclass PyarrowBatchConverter(BatchConverter):\n  def __init__(self, element_type: RowTypeConstraint):\n    super().__init__(pa.Table, element_type)\n    self._beam_schema = typing_to_runner_api(element_type).row_type.schema\n    arrow_schema = arrow_schema_from_beam_schema(self._beam_schema)\n\n    self._arrow_schema = arrow_schema\n\n  @staticmethod\n  def from_typehints(element_type,\n                     batch_type) -> Optional['PyarrowBatchConverter']:\n    assert batch_type == pa.Table\n\n    if not isinstance(element_type, RowTypeConstraint):\n      element_type = RowTypeConstraint.from_user_type(element_type)\n      if element_type is None:\n        raise TypeError(\n            f\"Element type {element_type} must be compatible with Beam Schemas \"\n            \"(https://beam.apache.org/documentation/programming-guide/#schemas)\"\n            \" for batch type pa.Table.\")\n\n    return PyarrowBatchConverter(element_type)\n\n  def produce_batch(self, elements):\n    arrays = [\n        pa.array([getattr(el, name) for el in elements],\n                 type=self._arrow_schema.field(name).type)\n        for name, _ in self._element_type._fields\n    ]\n    return pa.Table.from_arrays(arrays, schema=self._arrow_schema)\n\n  def explode_batch(self, batch: pa.Table):\n    \"\"\"Convert an instance of B to Generator[E].\"\"\"\n    for row_values in zip(*batch.columns):\n      yield self._element_type.user_type(","sourceCodeStart":296,"sourceCodeEnd":332,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/typehints/arrow_type_compatibility.py#L296-L332","documentation":"PyarrowBatchConverter.from_typehints requires the element type to be a Beam-schema-compatible row type when batch_type is pa.Table. If RowTypeConstraint.from_user_type cannot derive a schema from the given element type, it raises TypeError pointing to the Beam Schemas docs.","triggerScenarios":"Calling create_pyarrow_batch_converter / from_typehints with batch_type=pa.Table and an element_type that isn't a schema-able class (no typed NamedTuple/dataclass annotation, plain dict or arbitrary class) that from_user_type cannot convert.","commonSituations":"Passing a plain dict or non-annotated class as element_type to a batchable DoFn; a class annotated with types arrow can't map; forgetting @dataclass or NamedTuple typing so schema inference fails.","solutions":["Use a schema-annotated type: NamedTuple with type hints or @dataclass fields matching Beam Schemas","Register the type with @beam.typehints.with_output_types(RowTypeConstraint...) or call element_type = RowTypeConstraint.from_user_type(T) yourself and check it's not None","Change element data to conform to Beam Schemas (typed rows) before batching"],"exampleFix":"// before\nconverter = PyarrowBatchConverter.from_typehints(dict, pa.Table)\n// after\nclass Row(typing.NamedTuple):\n    x: int\n    y: str\nconverter = PyarrowBatchConverter.from_typehints(Row, pa.Table)","handlingStrategy":"validation","validationCode":"from apache_beam.typehints.row_type import RowTypeConstraint\nif RowTypeConstraint.from_user_type(element_type) is None:\n    raise TypeError(f'{element_type} is not Beam-Schema compatible; use a typed NamedTuple/dataclass')","typeGuard":"def is_schema_compatible(t) -> bool:\n    from apache_beam.typehints.row_type import RowTypeConstraint\n    return isinstance(t, RowTypeConstraint) or RowTypeConstraint.from_user_type(t) is not None","tryCatchPattern":"try:\n    converter = create_pyarrow_batch_converter(element_type, pa.Table)\nexcept TypeError:\n    converter = create_pyarrow_batch_converter(schema_annotated_row_type, pa.Table)","preventionTips":["Define batch element types as typed NamedTuple or @dataclass classes","Call beam.schema_inference to verify types map to Beam Schemas","Avoid plain dicts/untyped classes as batch element types"],"tags":["python","apache-beam","pyarrow","type-hints"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}