apache/beam · error · ValueError

partial and use_subprocess are mutually incompatible.

Error message

partial and use_subprocess are mutually incompatible.

What it means

apache_beam raises this ValueError in the DoFn/ParDo constructor when a user requests a 'partial' DoFn wrapper together with 'use_subprocess'. The two options describe contradictory process models (partially-evaluated function vs a function executed in a subprocess pool), so the library refuses to construct such a DoFn.

Source

Thrown at sdks/python/apache_beam/transforms/core.py:2354

      self,
      fn,
      args,
      kwargs,
      main_tag,
      dead_letter_tag,
      exc_class,
      partial,
      use_subprocess,
      threshold,
      threshold_windowing,
      timeout,
      error_handler,
      on_failure_callback,
      allow_unsafe_userstate_in_process,
      resource_hints,
      pardo_type_hints=None):
    if partial and use_subprocess:
      raise ValueError('partial and use_subprocess are mutually incompatible.')
    self._fn = fn
    self._args = args
    self._kwargs = kwargs
    self._main_tag = main_tag
    self._dead_letter_tag = dead_letter_tag
    self._exc_class = exc_class
    self._partial = partial
    self._use_subprocess = use_subprocess
    self._threshold = threshold
    self._threshold_windowing = threshold_windowing
    self._timeout = timeout
    self._error_handler = error_handler
    self._on_failure_callback = on_failure_callback
    self._allow_unsafe_userstate_in_process = allow_unsafe_userstate_in_process
    self._resource_hints = resource_hints
    self._pardo_type_hints = pardo_type_hints
    self._extra_tags = None

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove one of the two flags: set use_subprocess=False if you need partial=True
2. If subprocess isolation is required, drop partial=True and use a full DoFn
3. Refactor so the partial evaluation and the subprocess execution happen in separate ParDo transforms

Example fix

// before
ParDo(fn, partial=True, use_subprocess=True)
// after
ParDo(fn, use_subprocess=True)  # or partial=True without use_subprocess

if isinstance(pardo_kwargs.get('partial'), bool) and isinstance(pardo_kwargs.get('use_subprocess'), bool) and pardo_kwargs['partial'] and pardo_kwargs['use_subprocess']:
    raise ValueError('partial and use_subprocess cannot both be True')

def is_valid_pardo_config(kwargs: dict) -> bool:
    return not (kwargs.get('partial') and kwargs.get('use_subprocess'))

When it happens

Trigger: Calling ParDo(fn, ..., partial=True, use_subprocess=True) or passing equivalent flags through pardo/Map/FlatMap wrappers so that both partial=True and use_subprocess=True reach core.py's DoFn __init__.

Common situations: Users combining Beam's subprocess exception-handling feature (added for isolating crashing user code) with pipeline code that also uses partial DoFn evaluation, typically when merging two pipeline snippets with different options.

Related errors

• Transform '{full_label}' expects a PCollection as input. Got
• Returning a %s from a ParDo or FlatMap is not allowed. Pleas
• ParDo must be called with a DoFn instance.
• Main output tag %r must be different from side output tags %
• allow_unsafe_userstate_in_process is incompatible with excep

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e3f82b475a385cca. Report an issue: GitHub.