What causes "batch type must be pa.Table or pa.Array" in apache/beam?

Use batch_type=pa.Table (whole table batches) or pa.Array (column batches)

apache/beam · error · TypeError

batch type must be pa.Table or pa.Array

Error message

batch type must be pa.Table or pa.Array

What it means

create_pyarrow_batch_converter dispatches on batch_type: pa.Table routes to PyarrowBatchConverter and pa.Array to PyarrowArrayBatchConverter. Any other batch_type falls through and raises TypeError stating batch type must be pa.Table or pa.Array.

Solutions

Use batch_type=pa.Table (whole table batches) or pa.Array (column batches)
If you intended RecordBatch/DataFrame batches, use the corresponding converter (pandas_batch_converters or refactor to pa.Table)
Check the batchable decorator annotations so batch_type resolves to one of the two supported pyarrow types

Example fix

// before
@with_output_types(List[Row], batch_type=pa.RecordBatch)
// after
@with_output_types(List[Row], batch_type=pa.Table)

Defensive patterns

Strategy: validation

Validate before calling

import pyarrow as pa
if batch_type not in (pa.Table, pa.Array):
    raise TypeError(f'batch_type must be pa.Table or pa.Array, got {batch_type!r}')

Type guard

def valid_batch_type(bt) -> bool:
    import pyarrow as pa
    return bt in (pa.Table, pa.Array)

Try / catch

try:
    converter = create_pyarrow_batch_converter(element_type, batch_type)
except TypeError as e:
    if 'batch type must be' in str(e):
        converter = create_pyarrow_batch_converter(element_type, pa.Table)
    else:
        raise

Prevention

Only annotate batchable DoFns with pa.Table or pa.Array batch types
Use pandas batch converters if you actually want DataFrame batches
Centralize batch_type constants to avoid typos like pa.RecordBatch

When it happens

Trigger: Calling create_pyarrow_batch_converter(element_type, batch_type) with anything other than pa.Table or pa.Array — e.g. pa.RecordBatch, pa.DataFrame, pandas DataFrame, or a string/None batch type from a mismatched batchable annotation.

Common situations: Typing a batchable DoFn with @beam.typehints.with_input_types(...batch_type=pa.RecordBatch) instead of pa.Table/pa.Array; confusing pandas DataFrame batches with pyarrow batches; passing pa.Table subclasses or sentinel values.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/304c6579f4b0b258. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/typehints/arrow_type_compatibility.py:401

  def get_length(self, batch: pa.Array):
    return batch.num_rows

  def estimate_byte_size(self, batch: pa.Array):
    return batch.nbytes


@BatchConverter.register(name="pyarrow")
def create_pyarrow_batch_converter(
    element_type: type, batch_type: type) -> BatchConverter:
  if batch_type == pa.Table:
    return PyarrowBatchConverter.from_typehints(
        element_type=element_type, batch_type=batch_type)
  elif batch_type == pa.Array:
    return PyarrowArrayBatchConverter.from_typehints(
        element_type=element_type, batch_type=batch_type)

  raise TypeError("batch type must be pa.Table or pa.Array")

View on GitHub (pinned to 12126d8942)