apache/beam · error · ValueError

Scalar expression %s of type %s partitoned by non-singleton

Error message

Scalar expression %s of type %s partitoned by non-singleton %s

What it means

DeferredFrame.wrap only knows proxy types it has registered in _pandas_type_map; for an unregistered scalar-ish proxy type it falls back to _DeferredScalar, which is only valid for expressions partitioned by Singleton. If the expression's requires_partition_by() is not Singleton, wrapping as a scalar would silently mis-partition data, so a ValueError is raised.

Source

Thrown at sdks/python/apache_beam/dataframe/frame_base.py:70

  def wrap(cls, expr, split_tuples=True):
    proxy_type = type(expr.proxy())
    if proxy_type is tuple and split_tuples:

      def get(ix):
        return expressions.ComputedExpression(
            # yapf: disable
            'get_%d' % ix,
            lambda t: t[ix],
            [expr],
            requires_partition_by=partitionings.Arbitrary(),
            preserves_partition_by=partitionings.Singleton())

      return tuple(cls.wrap(get(ix)) for ix in range(len(expr.proxy())))
    elif proxy_type in cls._pandas_type_map:
      wrapper_type = cls._pandas_type_map[proxy_type]
    else:
      if expr.requires_partition_by() != partitionings.Singleton():
        raise ValueError(
            'Scalar expression %s of type %s partitoned by non-singleton %s' %
            (expr, proxy_type, expr.requires_partition_by()))
      wrapper_type = _DeferredScalar
    return wrapper_type(expr)

  def _elementwise(
      self, func, name=None, other_args=(), other_kwargs=None, inplace=False):
    other_kwargs = other_kwargs or {}
    return _elementwise_function(
        func, name, inplace=inplace)(self, *other_args, **other_kwargs)

  def __reduce__(self):
    return UnusableUnpickledDeferredBase, (str(self), )


class UnusableUnpickledDeferredBase(object):
  """Placeholder object used to break the transitive pickling chain in case a
  DeferredBase accidentially gets pickled (e.g. as part of globals).

View on GitHub (pinned to 12126d8942)

Solutions

Ensure the expression is elementwise (requires Singleton partitioning) before wrapping it as a scalar.
Reshape the result to a supported pandas type (DataFrame/Series) which wrap handles via its type map.
Use convert.to_pcollection/to_dataframe on the expression instead of wrapping it manually.

Example fix

// before
wrapper = frame_base.DeferredFrame.wrap(expr)  # expr returns unregistered type
// after
result = convert.to_pcolumn_like(expr)  # or cast proxy to pd.Series first
proxy = pd.Series(dtype=expr.proxy().dtype)
wrapper = frame_base.DeferredFrame.wrap(expressions.Bind(expr.func, proxy))

Defensive patterns

Strategy: validation

Validate before calling

from apache_beam.dataframe.partitionings import Singleton
if expr.requires_partition_by() != Singleton():
    raise ValueError('cannot wrap non-singleton expression as scalar')

Type guard

def wrappable_as_scalar(expr) -> bool:
    from apache_beam.dataframe.partitionings import Singleton
    return (type(expr.proxy()) in frame_base.DeferredFrame._pandas_type_map
            or expr.requires_partition_by() == Singleton())

Try / catch

try:
    frame = DeferredFrame.wrap(expr)
except ValueError as e:
    if 'non-singleton' in str(e):
        expr = make_elementwise_equivalent(expr)
        frame = DeferredFrame.wrap(expr)
    else:
        raise

Prevention

Only manually wrap expressions whose partitioning requirement is Singleton.
Prefer public convert.to_dataframe/to_pcollection over internal wrap calls.
Cast unusual proxy results (numpy scalars/arrays) into pd.Series/DataFrame before wrapping.

When it happens

Trigger: Wrapping an expression whose proxy is a custom/numpy scalar type not in _pandas_type_map while the expression requires non-singleton partitioning — typically from calling frame_base.DeferredFrame.wrap on a user-built expression or a projection producing an unusual type.

Common situations: Advanced/library code building custom expressions over dataframes; operations returning proxies of unusual types (e.g. numpy arrays) that need shuffling; hitting this after a pandas upgrade changes an inferred result type.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e56036cc556fba5e. Report an issue: GitHub.