apache/beam · error · WontImplementError

pivot() is not supported when pandas<1.4 and index is a…

Error message

pivot() is not supported when pandas<1.4 and index is a MultiIndex

What it means

Beam's DataFrame.pivot throws WontImplementError when pandas is older than 1.4 and a list-like index with more than one element is given, because pivot with a MultiIndex index requires pandas>=1.4 behavior that earlier versions lack.

Solutions

Upgrade pandas to >= 1.4 (pip install -U 'pandas>=1.4').
Use a single index column instead of a MultiIndex index.
Restructure via set_index on the deferred frame before pivoting on one column.
Fall back to local pandas with newer version for this operation.

Example fix

// before
result = df.pivot(index=['a', 'b'], columns='c')  # pandas 1.3
// after
pip install 'pandas>=1.4'
result = df.pivot(index=['a', 'b'], columns='c')

Defensive patterns

Strategy: validation

Validate before calling

import pandas as pd
def check_pivot_version(index):
    if tuple(map(int, pd.__version__.split('.')[:2])) < (1, 4) and isinstance(index, (list, tuple)) and len(index) > 1:
        raise ValueError('pivot with MultiIndex index requires pandas>=1.4')

Type guard

def pandas_at_least_1_4() -> bool:
    import pandas as pd
    return tuple(int(x) for x in pd.__version__.split('.')[:2]) >= (1, 4)

Try / catch

from apache_beam.dataframe import frame_base
try:
    result = df.pivot(index=['a', 'b'], columns='c')
except frame_base.WontImplementError:
    result = df.pivot(index='a', columns='c')

Prevention

Pin pandas>=1.4 in requirements for Beam DataFrame workloads.
Prefer single-column pivot indices.
Log the pandas version at pipeline startup to catch env drift.

When it happens

Trigger: Calling df.pivot(index=['a', 'b'], ...) on a DeferredDataFrame while the installed pandas version is < 1.4.

Common situations: Running Beam on an environment with an old pinned pandas version; upgrading pandas resolves it.

Understand the failure class

Background: "unsupported platform" / "not supported on this platform" errors: what they mean and how to fix them — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ccbbcadbf61bf5d9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:3889

        col_index = pd.MultiIndex.from_product(
          [values_in_col_index, *categories],
          names=names
        )
      else:
        # If one value provided, don't create a None level
        names = columns
        categories = [
          c.categories.astype('category') for c in selected_cols.dtypes
        ]
        col_index = pd.MultiIndex.from_product(
          categories,
          names=names
        )

    # Construct row index
    if index:
      if PD_VERSION < (1, 4) and is_list_like(index) and len(index) > 1:
        raise frame_base.WontImplementError(
          "pivot() is not supported when pandas<1.4 and index is a MultiIndex")
      per_partition = expressions.ComputedExpression(
          'pivot-per-partition',
          lambda df: df.set_index(keys=index), [self._expr],
          preserves_partition_by=partitionings.Singleton(),
          requires_partition_by=partitionings.Arbitrary()
      )
      tmp = per_partition.proxy().pivot(
        columns=columns, values=values, **kwargs)
      row_index = tmp.index
    else:
      per_partition = self._expr
      row_index = self._expr.proxy().index
    if PD_VERSION < (1, 4) and isinstance(row_index, pd.MultiIndex):
      raise frame_base.WontImplementError(
        "pivot() is not supported when pandas<1.4 and index is a MultiIndex")

    selected_values = self._expr.proxy()[values]

View on GitHub (pinned to 12126d8942)