What causes "batch type must be pd.Series or pd.DataFrame" in apache/beam?

Import pandas and pass the class object itself: batch_type=pd.DataFrame or batch_type=pd.Series.

apache/beam · error · TypeError

batch type must be pd.Series or pd.DataFrame

Error message

batch type must be pd.Series or pd.DataFrame

What it means

In Beam's pandas batch support, create_pandas_batch_converter builds a BatchConverter that maps between individual elements and pandas batches. Only pd.DataFrame and pd.Series are valid batch types; anything else raises this TypeError. The library deliberately restricts batching to these two pandas containers.

Source

Thrown at sdks/python/apache_beam/typehints/pandas_type_compatibility.py:149

  if fieldtype is not None:
    return fieldtype
  elif dtype.kind == 'S':
    return bytes
  else:
    return Any


@BatchConverter.register(name="pandas")
def create_pandas_batch_converter(
    element_type: type, batch_type: type) -> BatchConverter:
  if batch_type == pd.DataFrame:
    return DataFrameBatchConverter.from_typehints(
        element_type=element_type, batch_type=batch_type)
  elif batch_type == pd.Series:
    return SeriesBatchConverter.from_typehints(
        element_type=element_type, batch_type=batch_type)

  raise TypeError("batch type must be pd.Series or pd.DataFrame")


class DataFrameBatchConverter(BatchConverter):
  def __init__(
      self,
      element_type: RowTypeConstraint,
  ):
    super().__init__(pd.DataFrame, element_type)
    self._columns = [name for name, _ in element_type._fields]

  @staticmethod
  def from_typehints(element_type,
                     batch_type) -> Optional['DataFrameBatchConverter']:
    assert batch_type == pd.DataFrame

    if not isinstance(element_type, RowTypeConstraint):
      element_type = RowTypeConstraint.from_user_type(element_type)
      if element_type is None:

View on GitHub (pinned to 12126d8942)

Solutions

Import pandas and pass the class object itself: batch_type=pd.DataFrame or batch_type=pd.Series.
Check for string/config-driven batch types and resolve them to the actual pandas class before calling.
If you need another container (e.g. numpy arrays), use the torch/arrow converters or write a custom BatchConverter subclass instead.
Catch TypeError at converter construction to fail fast with a clearer pipeline error.

Example fix

// before
converter = create_pandas_batch_converter(element_type=element_type, batch_type='pd.DataFrame')
// after
import pandas as pd
converter = create_pandas_batch_converter(element_type=element_type, batch_type=pd.DataFrame)

Defensive patterns

Strategy: validation

Validate before calling

import pandas as pd
def valid_batch_type(batch_type) -> bool:
    return batch_type in (pd.DataFrame, pd.Series)

Type guard

def is_pandas_batch_type(batch_type) -> bool:
    import pandas as pd
    return batch_type in (pd.DataFrame, pd.Series)

Try / catch

try:
    converter = create_pandas_batch_converter(element_type=et, batch_type=bt)
except TypeError as e:
    raise ValueError(f'Unsupported batch type {bt!r}; use pd.DataFrame or pd.Series') from e

Prevention

Always pass pandas class objects, never strings or names from config.
Check the batch type table before wiring BatchElements into a pipeline.
Use pandas classes imported at module scope to avoid None placeholders.
Write a smoke test constructing the converter at pipeline-build time.

When it happens

Trigger: Passing batch_type to create_pandas_batch_converter (directly or through BatchConverter.from_typehints or the @with_batch_types / _unbatch_transform path) as something other than pd.DataFrame or pd.Series, e.g. a string 'DataFrame', a subclass, numpy.ndarray, or None.

Common situations: Configuring beam.BatchElements with a custom batch type, or wiring unbatch transforms in a pipeline where the batch type was typo'd or taken from a config instead of the actual pandas class object.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1183b472cd691d4d. Report an issue: GitHub.