apache/beam · error · WontImplementError

pivot() of non-categorical type is not supported because…

Error message

pivot() of non-categorical type is not supported because the type of the output column depends on the data. Please use pd.CategoricalDtype with explicit categories.

What it means

Beam's DataFrame.pivot throws WontImplementError when the values (or index) columns are not of CategoricalDtype, because the set of output columns would depend on the data encountered at runtime, violating Beam's requirement of a statically-known schema. Explicit pd.CategoricalDtype categories make the output columns known up front.

Solutions

Cast the relevant columns to pd.CategoricalDtype with an explicit categories list before pivoting.
Restrict values to columns already typed as categorical.
Use groupby-agg plus an explicit reshape (e.g. unstack on known keys) instead.
Fall back to local pandas if categories are not known in advance.

Example fix

// before
df['k'] = df['k'].astype('object')
result = df.pivot(index='i', columns='k')
// after
df['k'] = df['k'].astype(pd.CategoricalDtype(categories=['a', 'b', 'c']))
result = df.pivot(index='i', columns='k')

Defensive patterns

Strategy: validation

Validate before calling

def check_pivot_dtypes(df, index, values):
    cols = list(values or [c for c in df.columns if c not in (index,)])
    bad = [c for c in cols if not isinstance(df[c].dtype, pd.CategoricalDtype)]
    if bad:
        raise ValueError(f'Non-categorical pivot columns: {bad}')

Type guard

def is_categorical(col_dtype) -> bool:
    return isinstance(col_dtype, pd.CategoricalDtype)

Try / catch

from apache_beam.dataframe import frame_base
try:
    result = df.pivot(index='i', columns='k')
except frame_base.WontImplementError:
    df = df.assign(k=df['k'].astype(pd.CategoricalDtype(categories=['a', 'b'])))
    result = df.pivot(index='i', columns='k')

Prevention

Declare pivot keys as pd.CategoricalDtype with explicit categories.
Enumerate expected pivot columns at pipeline-construction time.
Avoid pivoting free-form string columns in distributed pipelines.

When it happens

Trigger: Calling df.pivot(...) on a DeferredDataFrame where index/values columns have plain (non-categorical) dtypes.

Common situations: Pivoting string-keyed data straight from pandas code; users unaware Beam needs pre-declared pivot categories.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/709ea57a41622bc3. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:3822

  count = _agg_method(pd.DataFrame, 'count')
  describe = _agg_method(pd.DataFrame, 'describe')
  max = _agg_method(pd.DataFrame, 'max')
  min = _agg_method(pd.DataFrame, 'min')

  @frame_base.with_docs_from(pd.DataFrame)
  @frame_base.args_to_kwargs(pd.DataFrame)
  @frame_base.populate_defaults(pd.DataFrame)
  def pivot(self, index=None, columns=None, values=None, **kwargs):
    """Because pivot is a non-deferred method, any columns specified in
    ``columns`` must be CategoricalDType so we can determine the output column
    names."""

    def verify_all_categorical(all_cols_are_categorical):
      if not all_cols_are_categorical:
        message = "pivot() of non-categorical type is not supported because " \
            "the type of the output column depends on the data. Please use " \
            "pd.CategoricalDtype with explicit categories."
        raise frame_base.WontImplementError(
          message, reason="non-deferred-columns")

    # If values not provided, take all remaining columns of dataframe
    if not values:
      tmp = self._expr.proxy()
      if index:
        tmp = tmp.drop(index, axis=1)
      if columns:
        tmp = tmp.drop(columns, axis=1)
      values = tmp.columns.values

    # Construct column index
    if is_list_like(columns) and len(columns) <= 1:
      columns = columns[0]
    selected_cols = self._expr.proxy()[columns]
    if isinstance(selected_cols, pd.Series):
      all_cols_are_categorical = isinstance(
        selected_cols.dtype, pd.CategoricalDtype

View on GitHub (pinned to 12126d8942)