apache/beam · error · NotImplementedError

groupby(as_index=False)

Error message

groupby(as_index=False)

What it means

Beam's groupby() only supports grouping by index labels; as_index=False would require resetting the group keys into columns, which the distributed implementation does not support. It raises NotImplementedError immediately when as_index is falsy.

Source

Thrown at sdks/python/apache_beam/dataframe/frames.py:379

      return frame_base.DeferredFrame.wrap(
          expressions.ComputedExpression(
              'last', lambda df: df.sort_index().last(offset=offset),
              [per_partition],
              preserves_partition_by=partitionings.Arbitrary(),
              requires_partition_by=partitionings.Singleton()))

  @frame_base.with_docs_from(pd.DataFrame)
  @frame_base.args_to_kwargs(pd.DataFrame)
  @frame_base.populate_defaults(pd.DataFrame)
  def groupby(self, by, level, axis, as_index, group_keys, **kwargs):
    """``as_index`` must be ``True``.

    Aggregations grouping by a categorical column with ``observed=False`` set
    are not currently parallelizable
    (`Issue 21827 <https://github.com/apache/beam/issues/21827>`_).
    """
    if not as_index:
      raise NotImplementedError('groupby(as_index=False)')

    if axis in (1, 'columns'):
      return _DeferredGroupByCols(
          expressions.ComputedExpression(
              'groupbycols', lambda df: df.groupby(
                  by, axis=axis, group_keys=group_keys, **kwargs), [self._expr],
              requires_partition_by=partitionings.Arbitrary(),
              preserves_partition_by=partitionings.Arbitrary()),
          group_keys=group_keys)

    if level is None and by is None:
      raise TypeError("You have to supply one of 'by' and 'level'")

    elif level is not None:
      if isinstance(level, (list, tuple)):
        grouping_indexes = level
      else:
        grouping_indexes = [level]

View on GitHub (pinned to 12126d8942)

Solutions

1. Keep as_index=True (default) and call reset_index() on the result to flatten group keys into columns
2. Restructure the pipeline so downstream consumers read group keys from the index
3. Run this portion with the pandas backend / convert to pandas if the data fits in memory

Example fix

# before
df.groupby('key', as_index=False).sum()
# after
df.groupby('key').sum().reset_index()

if not as_index:
    gb = df.groupby(by).agg(agg_fn)
    result = gb.reset_index()
else:
    result = df.groupby(by, as_index=True).agg(agg_fn)

try:
    out = df.groupby(by, as_index=False).agg(fn)
except NotImplementedError:
    out = df.groupby(by).agg(fn).reset_index()

When it happens

Trigger: df.groupby('col', as_index=False).mean() or df.groupby(['a','b'], as_index=False).agg(...) — any groupby call with as_index=False, including indirect calls from duplicated()/drop_duplicates()/aggregate().

Common situations: pandas users converting code where as_index=False produced flat DataFrames; aggregation reporting scripts expecting columns instead of a group-key index.

Related errors

• by
• Assigning an index is not yet supported. Consider using set_
• per-level align
• Indexing with a non-bool deferred frame is not yet supported
• set_index with Index or Series instances is not yet supporte

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/43800293fdf007b1. Report an issue: GitHub.