apache/beam · error · WontImplementError

unstack() is only supported on DataFrames if unstacked…

Error message

unstack() is only supported on DataFrames if unstacked level is a categorical or boolean column

What it means

For a DeferredDataFrame with a MultiIndex, unstack is only supported when every level being unstacked has CategoricalDtype or BooleanDtype. A non-categorical/boolean level would create result columns whose existence and order depend on the data, which Beam's deferred model cannot represent (reason "non-deferred-columns").

Solutions

Cast the level to categorical before unstacking: df.index = df.index.set_levels(df.index.levels[l].astype('category'), level=l).
Cast boolean-like levels to pandas BooleanDtype ('boolean').
If the dtype can't be categorical, do the unstack after to_pandas().

Example fix

// before
df.unstack(level='city')  # 'city' is object dtype -> WontImplementError
// after
df.index = df.index.set_levels(df.index.levels[df.index.names.index('city')].astype('category'), level='city')
df.unstack(level='city')

Defensive patterns

Strategy: validation

Validate before calling

import pandas as pd
level_numbers = [idx._get_level_number(l) for l in level_list]
if not all(isinstance(idx.levels[l].dtype, (pd.CategoricalDtype, pd.BooleanDtype)) for l in level_numbers):
    raise ValueError("unstack levels must be categorical or boolean dtype in Beam")

Type guard

def is_unstackable_level(level_dtype):
    return isinstance(level_dtype, (pd.CategoricalDtype, pd.BooleanDtype))

Try / catch

from apache_beam.dataframe import frame_base
try:
    out = df.unstack(level=levels)
except frame_base.WontImplementError:
    out = df.to_pandas().unstack(level=levels)

Prevention

Cast MultiIndex levels to 'category' (or 'boolean') dtype before unstacking.
Inspect index.dtypes of the deferred frame's proxy before planning an unstack.
Prefer joins/pivots on categorical keys when designing the pipeline.

When it happens

Trigger: Calling unstack() or unstack(level=...) on a DeferredDataFrame with a MultiIndex where at least one of the requested levels is a plain (object/int) dtype rather than categorical or boolean.

Common situations: Unstacking string or integer index levels migrated from pandas; forgetting to convert the level to a categorical dtype before the pipeline.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6ed6bdc062580d7b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:1005

            "unstack() is not supported when using pandas < 1.2.0\n"
            "Please upgrade to pandas 1.2.0 or higher to use this operation.")
      return frame_base.DeferredFrame.wrap(
          expressions.ComputedExpression(
              'unstack', lambda s: s.unstack(**kwargs), [self._expr],
              requires_partition_by=partitionings.Index()))
    else:
      # Unstacking MultiIndex objects
      idx = self._expr.proxy().index

      # Converting level (int, str, or combination) to a list of number levels
      level_list = level if isinstance(level, list) else [level]
      level_number_list = [idx._get_level_number(l) for l in level_list]

      # Checking if levels provided are of CategoricalDtype
      if not all(isinstance(idx.levels[l].dtype, (pd.CategoricalDtype,
                                                  pd.BooleanDtype))
                 for l in level_number_list):
        raise frame_base.WontImplementError(
            "unstack() is only supported on DataFrames if unstacked level "
            "is a categorical or boolean column",
            reason="non-deferred-columns")
      else:
        tmp = self._expr.proxy().unstack(**kwargs)
        if isinstance(tmp.columns, pd.MultiIndex):
          levels = []
          for i in range(tmp.columns.nlevels):
            level = tmp.columns.levels[i]
            levels.append(level)
          col_idx = pd.MultiIndex.from_product(levels)
        else:
          if tmp.columns.dtype == 'boolean':
            col_idx = pd.Index([False, True], dtype='boolean')
          else:
            col_idx = pd.CategoricalIndex(tmp.columns.categories)

        if isinstance(self._expr.proxy(), pd.Series):

View on GitHub (pinned to 12126d8942)