apache/beam · error · TypeError

Testing the truth value of a deferred scalar is not allowed.

Error message

Testing the truth value of a deferred scalar is not allowed. It's not possible to branch on the result of deferred operations.

What it means

_DeferredScalar.__bool__ deliberately raises TypeError: deferred scalars represent results computed at pipeline runtime, so their truth value is unknown at graph-construction time. Branching (if/and/or/not) on such a value would require materializing pipeline data eagerly, which the DataFrame API forbids.

Source

Thrown at sdks/python/apache_beam/dataframe/frame_base.py:130

              name,
              func, [self._expr] + [arg._expr for arg in args],
              requires_partition_by=partitionings.Singleton()))

  def __neg__(self):
    return self.apply(operator.neg)

  def __pos__(self):
    return self.apply(operator.pos)

  def __invert__(self):
    return self.apply(operator.invert)

  def __repr__(self):
    return f"DeferredScalar[type={type(self._expr.proxy())}]"

  def __bool__(self):
    # TODO(BEAM-11951): Link to documentation
    raise TypeError(
        "Testing the truth value of a deferred scalar is not "
        "allowed. It's not possible to branch on the result of "
        "deferred operations.")


def _scalar_binop(op):
  def binop(self, other):
    if not isinstance(other, DeferredBase):
      return self.apply(lambda left: getattr(left, op)(other), name=op)
    elif isinstance(other, _DeferredScalar):
      return self.apply(
          lambda left, right: getattr(left, op)(right), name=op, args=[other])
    else:
      return NotImplemented

  return binop

View on GitHub (pinned to 12126d8942)

Solutions

Restructure the pipeline to express the branch data-parallel, e.g. with filter/map and boolean column arithmetic instead of Python if.
Convert to_pcollection / to a concrete value first (materialize) and branch outside the deferred API.
Reorder so the condition is on concrete Python data known at graph-build time.

Example fix

// before
if df['x'].sum() > 0:
  df = df.filter(...)
// after
df = df[df['x'].sum() > 0]  # elementwise filter, no Python branch
# or materialize first:
total = convert.to_pcollection(df[['x']].sum(), pipeline=pipeline)

Defensive patterns

Strategy: type-guard

Validate before calling

from apache_beam.dataframe.frame_base import DeferredScalar
if isinstance(value, DeferredScalar):
    raise TypeError('do not branch on deferred scalars; restructure with filter/map')

Type guard

def is_deferred_scalar(v) -> bool:
    from apache_beam.dataframe.frame_base import DeferredScalar
    return isinstance(v, DeferredScalar)

Prevention

Treat every value derived from a deferred dataframe as graph-time opaque; never use it in if/and/or.
Express conditionals as boolean column arithmetic and df[bool_expr] filtering.
Materialize (to_pcollection + run) when you truly need a Python-level branch.

When it happens

Trigger: Writing `if deferred_scalar:` or using a deferred scalar in boolean context (not, and, or, assert, any condition) inside a Beam dataframe pipeline.

Common situations: Porting pandas code that branches on computed results; validating a computed value with `if df.max().x > 0:`; novices treating deferred objects like concrete Python values during pipeline construction.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b8846f06f474468c. Report an issue: GitHub.