{"record":{"id":"836039c3c1449c14","repo":"apache/beam","slug":"dofn-self-r-yields-batches-from-both-process-and-process","errorCode":null,"errorMessage":"DoFn {self!r} yields batches from both process and process_batch, but they produce different types:\n process: {output_batch_type}\n process_batch: {process_batch_type!r}","messagePattern":"DoFn (.+?) yields batches from both process and process_batch, but they produce different types:\n process: (.+?)\n process_batch: (.+?)","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/transforms/core.py","lineNumber":964,"sourceCode":"\n    Returns:\n      ``None`` if this DoFn will never yield batches, else a Beam typehint or\n      a native Python typehint.\n    \"\"\"\n    output_batch_type = None\n    if self._process_defined and self._process_yields_batches:\n      output_batch_type = self._get_element_type_from_return_annotation(\n          self.process, input_element_type)\n    if self._process_batch_defined and not self._process_batch_yields_elements:\n      process_batch_type = self._get_element_type_from_return_annotation(\n          self.process_batch,\n          self._get_input_batch_type_normalized(input_element_type))\n\n      # TODO: Consider requiring an inheritance relationship rather than\n      # equality\n      if (output_batch_type is not None and\n          (not process_batch_type == output_batch_type)):\n        raise TypeError(\n            f\"DoFn {self!r} yields batches from both process and \"\n            \"process_batch, but they produce different types:\\n\"\n            f\" process: {output_batch_type}\\n\"\n            f\" process_batch: {process_batch_type!r}\")\n\n      output_batch_type = process_batch_type\n\n    return output_batch_type\n\n  def _process_argspec_fn(self):\n    \"\"\"Returns the Python callable that will eventually be invoked.\n\n    This should ideally be the user-level function that is called with\n    the main and (if any) side inputs, and is used to relate the type\n    hint parameters with the input parameters (e.g., by argument name).\n    \"\"\"\n    return self.process\n","sourceCodeStart":946,"sourceCodeEnd":982,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/transforms/core.py#L946-L982","documentation":"When a DoFn defines both `process` and `process_batch` that both yield batches, get_output_batch_type (core.py:964) checks the two declared batch output types are equal. Different batch types would make the output type ambiguous, so it raises this TypeError.","triggerScenarios":"A DoFn where process's output batch type (e.g. List[str]) differs from process_batch's (e.g. pandas.DataFrame or List[int]), both non-None and unequal.","commonSituations":"Incrementally adding a batched path to an existing DoFn; one method annotated with element lists and the other with DataFrame/Table batch types.","solutions":["Align the batch output types of `process` and `process_batch` so they produce the same batch type.","Remove one of the two methods so only a single batch-producing implementation exists.","If `process` should yield elements, not batches, remove @yields_batches / adjust annotations so it is treated as element-wise."],"exampleFix":"# before\nclass MyDoFn(DoFn):\n    def process(self, x) -> Iterator[List[int]]: ...\n    def process_batch(self, batch) -> Iterator[pd.DataFrame]: ...\n\n# after\nclass MyDoFn(DoFn):\n    def process_batch(self, batch) -> Iterator[pd.DataFrame]: ...","handlingStrategy":"validation","validationCode":"def check_batch_types_match(dofn):\n    if hasattr(dofn, 'process') and hasattr(dofn, 'process_batch'):\n        pt = dofn.get_output_batch_type()\n        # ensure process and process_batch declare identical batch types\n        if pt is not None and getattr(dofn, '_declared_process_batch_type', pt) != pt:\n            raise TypeError('process and process_batch batch output types differ')","typeGuard":"def single_batch_path(cls) -> bool:\n    return not (hasattr(cls, 'process') and hasattr(cls, 'process_batch'))","tryCatchPattern":"try:\n    bt = dofn.get_output_batch_type()\nexcept TypeError as e:\n    if 'produce different types' in str(e):\n        logger.error('Unify batch output types across process/process_batch: %s', e)\n    raise","preventionTips":["Keep only one batch-producing method (process_batch) per DoFn","Use identical annotations for both methods if you must keep both","Add type-hint consistency tests for DoFns"],"tags":["python","apache-beam","type-hints","typeerror"],"backgroundTag":"type-mismatch","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}