apache/beam · error · ValueError

CombineFn.setup and CombineFn.teardown are not supported wit

Error message

CombineFn.setup and CombineFn.teardown are not supported with non-portable Dataflow runner. Please use Dataflow Portable Runner instead.

What it means

The non-portable (legacy) Dataflow runner never invokes CombineFn.setup()/teardown(), so if a pipeline uses a CombineFn that overrides them the DataflowRunner raises this ValueError during translation instead of silently ignoring the lifecycle hooks. Dataflow Runner v2 (portable) supports them.

Source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_runner.py:330

    return FlattenInputVisitor()

  @staticmethod
  def combinefn_visitor():
    # Imported here to avoid circular dependencies.
    from apache_beam import core
    from apache_beam.pipeline import PipelineVisitor

    class CombineFnVisitor(PipelineVisitor):
      """Checks if `CombineFn` has non-default setup or teardown methods.
      If yes, raises `ValueError`.
      """
      def visit_transform(self, applied_transform):
        transform = applied_transform.transform
        if isinstance(transform, core.ParDo) and isinstance(
            transform.fn, core.CombineValuesDoFn):
          if self._overrides_setup_or_teardown(transform.fn.combinefn):
            raise ValueError(
                'CombineFn.setup and CombineFn.teardown are '
                'not supported with non-portable Dataflow '
                'runner. Please use Dataflow Portable Runner instead.')

      @staticmethod
      def _overrides_setup_or_teardown(combinefn):
        # TODO(https://github.com/apache/beam/issues/18716): provide an
        # implementation for this method
        return False

    return CombineFnVisitor()

  def _adjust_pipeline_for_dataflow_v2(self, pipeline):
    # Dataflow runner requires a KV type for GBK inputs, hence we enforce that
    # here.
    pipeline.visit(
        group_by_key_input_visitor(
            not pipeline._options.view_as(

View on GitHub (pinned to 12126d8942)

Solutions

1. Run on Dataflow Runner v2: remove any runner-v2-disabling experiments (do not set --dataflow_runner_v2=False or the disable experiment).
2. Move the setup/teardown logic into create_accumulator()/add_input()/extract_output() or into the DoFn's setup/teardown of a ParDo.
3. Upgrade apache-beam to a version where Runner v2 is the default for Dataflow.
4. Refactor the CombineFn so its state is self-contained and needs no per-bundle lifecycle hooks.

Example fix

// before
options.view_as(TypeOptions).experiments = ['disable_runner_v2']
// after
# remove disable_runner_v2 experiment; Runner v2 supports CombineFn.setup/teardown

def combinefn_needs_runner_v2(fn):
    import inspect
    base = apache_beam.transforms.combinefn.CombineFn
    return (type(fn).setup is not base.setup or type(fn).teardown is not base.teardown)

try:
    pipeline.run()
except ValueError as e:
    if 'CombineFn.setup' in str(e):
        raise SystemExit('Remove disable_runner_v2 experiment or refactor CombineFn')

When it happens

Trigger: Running a pipeline on the legacy Dataflow runner whose CombineFn overrides setup() or teardown(); visit_transform detects core.CombineValuesDoFn whose combinefn overrides those methods while runner v2 is disabled.

Common situations: Combining custom resource setup (DB connections, file handles) inside a CombineFn and submitting to legacy Dataflow; pipelines written for portable runners ported back to the legacy runner.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Disabling Dataflow Portable Runner no longer supported using
• Could not find element
• Too many matches
• Could not translate the internal step name %r since job grap
• Could not translate the internal step name %r.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e6c22f9b534a1568. Report an issue: GitHub.