apache/beam · error · TypeError

DoFn {self!r} yields batches from both process and process_b

Error message

DoFn {self!r} yields batches from both process and process_batch, but they produce different types:
 process: {output_batch_type}
 process_batch: {process_batch_type!r}

What it means

When a DoFn defines both `process` and `process_batch` that both yield batches, get_output_batch_type (core.py:964) checks the two declared batch output types are equal. Different batch types would make the output type ambiguous, so it raises this TypeError.

Source

Thrown at sdks/python/apache_beam/transforms/core.py:964

    Returns:
      ``None`` if this DoFn will never yield batches, else a Beam typehint or
      a native Python typehint.
    """
    output_batch_type = None
    if self._process_defined and self._process_yields_batches:
      output_batch_type = self._get_element_type_from_return_annotation(
          self.process, input_element_type)
    if self._process_batch_defined and not self._process_batch_yields_elements:
      process_batch_type = self._get_element_type_from_return_annotation(
          self.process_batch,
          self._get_input_batch_type_normalized(input_element_type))

      # TODO: Consider requiring an inheritance relationship rather than
      # equality
      if (output_batch_type is not None and
          (not process_batch_type == output_batch_type)):
        raise TypeError(
            f"DoFn {self!r} yields batches from both process and "
            "process_batch, but they produce different types:\n"
            f" process: {output_batch_type}\n"
            f" process_batch: {process_batch_type!r}")

      output_batch_type = process_batch_type

    return output_batch_type

  def _process_argspec_fn(self):
    """Returns the Python callable that will eventually be invoked.

    This should ideally be the user-level function that is called with
    the main and (if any) side inputs, and is used to relate the type
    hint parameters with the input parameters (e.g., by argument name).
    """
    return self.process

View on GitHub (pinned to 12126d8942)

Solutions

Align the batch output types of `process` and `process_batch` so they produce the same batch type.
Remove one of the two methods so only a single batch-producing implementation exists.
If `process` should yield elements, not batches, remove @yields_batches / adjust annotations so it is treated as element-wise.

Example fix

# before
class MyDoFn(DoFn):
    def process(self, x) -> Iterator[List[int]]: ...
    def process_batch(self, batch) -> Iterator[pd.DataFrame]: ...

# after
class MyDoFn(DoFn):
    def process_batch(self, batch) -> Iterator[pd.DataFrame]: ...

Defensive patterns

Strategy: validation

Validate before calling

def check_batch_types_match(dofn):
    if hasattr(dofn, 'process') and hasattr(dofn, 'process_batch'):
        pt = dofn.get_output_batch_type()
        # ensure process and process_batch declare identical batch types
        if pt is not None and getattr(dofn, '_declared_process_batch_type', pt) != pt:
            raise TypeError('process and process_batch batch output types differ')

Type guard

def single_batch_path(cls) -> bool:
    return not (hasattr(cls, 'process') and hasattr(cls, 'process_batch'))

Try / catch

try:
    bt = dofn.get_output_batch_type()
except TypeError as e:
    if 'produce different types' in str(e):
        logger.error('Unify batch output types across process/process_batch: %s', e)
    raise

Prevention

Keep only one batch-producing method (process_batch) per DoFn
Use identical annotations for both methods if you must keep both
Add type-hint consistency tests for DoFns

When it happens

Trigger: A DoFn where process's output batch type (e.g. List[str]) differs from process_batch's (e.g. pandas.DataFrame or List[int]), both non-None and unequal.

Common situations: Incrementally adding a batched path to an existing DoFn; one method annotated with element lists and the other with DataFrame/Table batch types.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/836039c3c1449c14. Report an issue: GitHub.