apache/beam · error · Exception

allow_unsafe_userstate_in_process is incompatible with excep

Error message

allow_unsafe_userstate_in_process is incompatible with exception handling done with subprocesses or timeouts. If you need this feature, comment in https://github.com/apache/beam/issues/35976

What it means

apache_beam raises this Exception when _allow_unsafe_userstate_in_process is enabled on a DoFn whose exception handling uses subprocesses or timeouts. Such exception handling re-executes or isolates user code in a way that conflicts with reusing in-process user state, so the combination is explicitly unimplemented (tracked as Beam issue #35976).

Source

Thrown at sdks/python/apache_beam/transforms/core.py:2384

    self._error_handler = error_handler
    self._on_failure_callback = on_failure_callback
    self._allow_unsafe_userstate_in_process = allow_unsafe_userstate_in_process
    self._resource_hints = resource_hints
    self._pardo_type_hints = pardo_type_hints
    self._extra_tags = None

  def with_outputs(self, *tags, main=None):
    self._extra_tags = tags
    if main is not None:
      self._main_tag = main
    return self

  def _build_pardo(self, pcoll):
    """Build the inner ParDo with the exception-handling wrapper DoFn."""
    if self._allow_unsafe_userstate_in_process:
      if self._use_subprocess or self._timeout:
        # TODO(https://github.com/apache/beam/issues/35976): Implement this
        raise Exception(
            'allow_unsafe_userstate_in_process is incompatible with ' +
            'exception handling done with subprocesses or timeouts. If you ' +
            'need this feature, comment in ' +
            'https://github.com/apache/beam/issues/35976')
    if self._use_subprocess:
      wrapped_fn = _SubprocessDoFn(self._fn, timeout=self._timeout)
    elif self._timeout:
      wrapped_fn = _TimeoutDoFn(self._fn, timeout=self._timeout)
    else:
      wrapped_fn = self._fn
    pardo = ParDo(
        _ExceptionHandlingWrapperDoFn(
            wrapped_fn,
            self._dead_letter_tag,
            self._exc_class,
            self._partial,
            self._on_failure_callback,
            self._allow_unsafe_userstate_in_process,

View on GitHub (pinned to 12126d8942)

Solutions

Turn off allow_unsafe_userstate_in_process, or
Remove use_subprocess and timeout so exception handling stays in-process
Comment/upvote the feature request at https://github.com/apache/beam/issues/35976 if you need the combination

Example fix

// before
ParDo(fn, timeout=30, allow_unsafe_userstate_in_process=True)
// after
ParDo(fn, timeout=30)  # drop allow_unsafe_userstate_in_process

Defensive patterns

Strategy: validation

Validate before calling

if allow_unsafe_userstate_in_process and (use_subprocess or timeout is not None):
    raise ValueError('allow_unsafe_userstate_in_process cannot be combined with subprocess/timeout exception handling (beam#35976)')

Type guard

def compatible_exc_handling(unsafe_userstate: bool, use_subprocess: bool, timeout) -> bool:
    return not (unsafe_userstate and (use_subprocess or timeout is not None))

Try / catch

try:
    expand(pardo)
except Exception as e:
    if 'allow_unsafe_userstate_in_process is incompatible' in str(e):
        retry_without_unsafe_userstate()  # rebuild transform with the flag off
    else:
        raise

Prevention

Track beam issue 35976 before opting into unsafe user state
Keep exception-handling options in a single config object validated at pipeline-build time

When it happens

Trigger: Constructing a ParDo with allow_unsafe_userstate_in_process=True together with use_subprocess=True or a non-None timeout, then building the transform via _build_pardo (e.g. at pipeline expansion time).

Common situations: Developers enabling the unsafe-user-state opt-in for performance, while also using the exception-handling wrapper with timeouts or subprocess isolation, hit this at pipeline graph construction rather than runtime data processing.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/404ee8437c575eb5. Report an issue: GitHub.