apache/beam · error · TypeError

Failed to find a BatchConverter for the output types of…

Error message

Failed to find a BatchConverter for the *output* types of DoFn {self.fn!r} (element_type={output_element_type!r}, batch_type={output_batch_type!r}). Maybe you need to override DoFn.infer_output_type to set the output element type?

What it means

When the DoFn's output batch type is known, Beam builds an output BatchConverter from (output_element_type, output_batch_type). If no converter exists for that pair, the TypeError is re-raised with this message, hinting that infer_output_type may need overriding.

Solutions

Override DoFn.infer_output_type to return a concrete supported element type (e.g. np.int64, a registered row type).
Register a custom BatchConverter for the (element_type, batch_type) pair.
Change the process_batch return annotation to a supported batch type (np.ndarray, pandas.DataFrame, pa.Table).
Install pandas/pyarrow if the intended converter requires them.
Example fix: override infer_output_type to return numpy.int64 so the output converter can be created.

Example fix

// before
class MyDoFn(beam.DoFn):
    def process_batch(self, batch: np.ndarray) -> Iterator[np.ndarray]: ...
// after
class MyDoFn(beam.DoFn):
    def infer_output_type(self, input_element_type):
        return np.int64
    def process_batch(self, batch: np.ndarray) -> Iterator[np.ndarray]: ...

Defensive patterns

Strategy: validation

Validate before calling

out_t = MyDoFn().infer_output_type(el_t)
try:
    BatchConverter.from_typehints(element_type=out_t, batch_type=out_batch_t)
except TypeError:
    raise TypeError('override infer_output_type with a supported element type')

Type guard

def output_converter_exists(dofn, el_t) -> bool:
    try:
        BatchConverter.from_typehints(
            element_type=dofn.infer_output_type(el_t),
            batch_type=get_output_batch_type(dofn))
        return True
    except TypeError:
        return False

Try / catch

try:
    run_batch_pipeline()
except TypeError as e:
    if 'output' in str(e) and 'BatchConverter' in str(e):
        fix_infer_output_type(); run_batch_pipeline()
    else:
        raise

Prevention

Override DoFn.infer_output_type whenever outputs are batches of non-Any elements.
Keep output element and batch types aligned with registered converters.
Ensure pandas/pyarrow are installed for DataFrame/Table batch outputs.

When it happens

Trigger: Batch-yielding DoFn whose output element type (from infer_output_type, often defaulting to Any or an unsupported type) paired with the declared batch type has no registered BatchConverter.

Common situations: Custom element classes with pandas/arrow batch outputs; forgetting to override DoFn.infer_output_type so element type defaults incorrectly; missing optional deps (pandas/pyarrow) so no converter is registered.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0b6c0426b8dd2797. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/core.py:1752

    if self.fn._can_yield_batches:
      output_batch_type = self.fn._get_output_batch_type_normalized(
          input_element_type)
      if output_batch_type is None:
        # TODO: Mention process method in this error
        raise TypeError(
            f"process_batch method on {self.fn!r} does not have "
            "a return type annoation")

      # Generate a batch converter to convert between the output type and the
      # (batch) output type of process_batch
      output_element_type = self.infer_output_type(input_element_type)

      try:
        self.fn.output_batch_converter = BatchConverter.from_typehints(
            element_type=output_element_type, batch_type=output_batch_type)
      except TypeError as e:
        raise TypeError(
            "Failed to find a BatchConverter for the *output* types of DoFn "
            f"{self.fn!r} (element_type={output_element_type!r}, "
            f"batch_type={output_batch_type!r}). Maybe you need to override "
            "DoFn.infer_output_type to set the output element type?") from e
    else:
      self.fn.output_batch_converter = None

  def make_fn(self, fn, has_side_inputs):
    if isinstance(fn, DoFn):
      return fn
    return CallableWrapperDoFn(fn)

  def _process_argspec_fn(self):
    return self.fn._process_argspec_fn()

  def display_data(self):
    return {
        'fn': DisplayDataItem(self.fn.__class__, label='Transform Function'),

View on GitHub (pinned to 12126d8942)

Failed to find a BatchConverter for the *output* types of…