{"record":{"id":"304c6579f4b0b258","repo":"apache/beam","slug":"batch-type-must-be-pa-table-or-pa-array","errorCode":null,"errorMessage":"batch type must be pa.Table or pa.Array","messagePattern":"batch type must be pa\\.Table or pa\\.Array","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/typehints/arrow_type_compatibility.py","lineNumber":401,"sourceCode":"\n  def get_length(self, batch: pa.Array):\n    return batch.num_rows\n\n  def estimate_byte_size(self, batch: pa.Array):\n    return batch.nbytes\n\n\n@BatchConverter.register(name=\"pyarrow\")\ndef create_pyarrow_batch_converter(\n    element_type: type, batch_type: type) -> BatchConverter:\n  if batch_type == pa.Table:\n    return PyarrowBatchConverter.from_typehints(\n        element_type=element_type, batch_type=batch_type)\n  elif batch_type == pa.Array:\n    return PyarrowArrayBatchConverter.from_typehints(\n        element_type=element_type, batch_type=batch_type)\n\n  raise TypeError(\"batch type must be pa.Table or pa.Array\")\n","sourceCodeStart":383,"sourceCodeEnd":402,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/typehints/arrow_type_compatibility.py#L383-L402","documentation":"create_pyarrow_batch_converter dispatches on batch_type: pa.Table routes to PyarrowBatchConverter and pa.Array to PyarrowArrayBatchConverter. Any other batch_type falls through and raises TypeError stating batch type must be pa.Table or pa.Array.","triggerScenarios":"Calling create_pyarrow_batch_converter(element_type, batch_type) with anything other than pa.Table or pa.Array — e.g. pa.RecordBatch, pa.DataFrame, pandas DataFrame, or a string/None batch type from a mismatched batchable annotation.","commonSituations":"Typing a batchable DoFn with @beam.typehints.with_input_types(...batch_type=pa.RecordBatch) instead of pa.Table/pa.Array; confusing pandas DataFrame batches with pyarrow batches; passing pa.Table subclasses or sentinel values.","solutions":["Use batch_type=pa.Table (whole table batches) or pa.Array (column batches)","If you intended RecordBatch/DataFrame batches, use the corresponding converter (pandas_batch_converters or refactor to pa.Table)","Check the batchable decorator annotations so batch_type resolves to one of the two supported pyarrow types"],"exampleFix":"// before\n@with_output_types(List[Row], batch_type=pa.RecordBatch)\n// after\n@with_output_types(List[Row], batch_type=pa.Table)","handlingStrategy":"validation","validationCode":"import pyarrow as pa\nif batch_type not in (pa.Table, pa.Array):\n    raise TypeError(f'batch_type must be pa.Table or pa.Array, got {batch_type!r}')","typeGuard":"def valid_batch_type(bt) -> bool:\n    import pyarrow as pa\n    return bt in (pa.Table, pa.Array)","tryCatchPattern":"try:\n    converter = create_pyarrow_batch_converter(element_type, batch_type)\nexcept TypeError as e:\n    if 'batch type must be' in str(e):\n        converter = create_pyarrow_batch_converter(element_type, pa.Table)\n    else:\n        raise","preventionTips":["Only annotate batchable DoFns with pa.Table or pa.Array batch types","Use pandas batch converters if you actually want DataFrame batches","Centralize batch_type constants to avoid typos like pa.RecordBatch"],"tags":["python","apache-beam","pyarrow","batching"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}