apache/beam · error · TypeError

DoFn {self!r} yields element from both process and process_b

Error message

DoFn {self!r} yields element from both process and process_batch, but they have mismatched output typehints:
 process: {process_type_hints.output_types}
 process_batch: {process_batch_type_hints.output_types}

What it means

When a DoFn defines both `process` and `process_batch` and both declare output type hints, Beam requires them to agree (core.py:814). Mismatched output typehints make the pipeline's inferred output type ambiguous, so default_type_hints raises this TypeError.

Source

Thrown at sdks/python/apache_beam/transforms/core.py:814

      # process() produces batches, don't use it's output typehint
      process_type_hints = process_type_hints.with_output_types_from(
          typehints.decorators.IOTypeHints.empty())

    if self._process_batch_yields_elements:
      # process_batch() produces elements, *do* use it's output typehint

      # First access the typehint
      process_batch_type_hints = typehints.decorators.IOTypeHints.from_callable(
          self.process_batch) or typehints.decorators.IOTypeHints.empty()

      # Then we deconflict with the typehint from process, if it exists
      if (process_batch_type_hints.output_types
          != typehints.decorators.IOTypeHints.empty().output_types):
        if (process_type_hints.output_types
            != typehints.decorators.IOTypeHints.empty().output_types and
            process_batch_type_hints.output_types
            != process_type_hints.output_types):
          raise TypeError(
              f"DoFn {self!r} yields element from both process and "
              "process_batch, but they have mismatched output typehints:\n"
              f" process: {process_type_hints.output_types}\n"
              f" process_batch: {process_batch_type_hints.output_types}")

        process_type_hints = process_type_hints.with_output_types_from(
            process_batch_type_hints)

    try:
      process_type_hints = process_type_hints.strip_iterable()
    except ValueError as e:
      raise ValueError('Return value not iterable: %s: %s' % (self, e))
    process_type_hints = process_type_hints.extract_tagged_outputs()

    # Prefer class decorator type hints for backwards compatibility.
    return get_type_hints(self.__class__).with_defaults(process_type_hints)

  # TODO(sourabhbajaj): Do we want to remove the responsibility of these from

View on GitHub (pinned to 12126d8942)

Solutions

  1. Make the output type hints of `process` and `process_batch` consistent (same inner element type).
  2. Delete the now-redundant `process` method if you only use `process_batch`.
  3. Remove explicit output hints from one of the methods so only one source of truth exists.

Example fix

# before
class MyDoFn(DoFn):
    def process(self, x) -> Iterable[str]: ...
    def process_batch(self, batch) -> Iterator[List[int]]: ...

# after
class MyDoFn(DoFn):
    def process(self, x) -> Iterator[str]: ...
    def process_batch(self, batch) -> Iterator[List[str]]: ...
Defensive patterns

Strategy: validation

Validate before calling

import inspect
from typing import get_type_hints
def check_matching_output_hints(dofn_cls):
    if hasattr(dofn_cls, 'process') and hasattr(dofn_cls, 'process_batch'):
        p = get_type_hints(dofn_cls.process).get('return')
        pb = get_type_hints(dofn_cls.process_batch).get('return')
        if p is not None and pb is not None and p != pb:
            raise TypeError(f'process ({p}) and process_batch ({pb}) output hints differ')

Type guard

def has_consistent_hints(cls) -> bool:
    if not (hasattr(cls, 'process') and hasattr(cls, 'process_batch')):
        return True
    from typing import get_type_hints
    return get_type_hints(cls.process).get('return') in (None, get_type_hints(cls.process_batch).get('return'))

Try / catch

try:
    hints = dofn.default_type_hints()
except TypeError as e:
    if 'mismatched output typehints' in str(e):
        logger.error('Align process/process_batch output hints: %s', e)
    raise

Prevention

When it happens

Trigger: A DoFn subclass where `process` has e.g. @with_output_types(Iterable[str]) while `process_batch` is annotated/typed as returning batches of ints, and both output hint sets are non-empty and unequal.

Common situations: Migrating a DoFn from element-wise `process` to batched `process_batch` while leaving old type hints on `process`; annotations added incrementally by different contributors.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c8a0d9a6b153e9fc. Report an issue: GitHub.