apache/beam · error · NotImplementedError

dropna=False does not work as intended in the Beam…

Error message

dropna=False does not work as intended in the Beam DataFrame API when grouping on multiple columns or indexes (See https://github.com/apache/beam/issues/21014).

What it means

DeferredGroupBy raises NotImplementedError when dropna=False is combined with grouping on multiple columns/indexes (index.nlevels > 1), because Beam's grouping cannot keep NaN-key groups correctly in that case; see GitHub issue 21014.

Solutions

Fill NaN values in the grouping columns before grouping (e.g. fillna('__missing__')) and keep dropna default True
Group on a single column/index level when using dropna=False
Drop NaN-key rows explicitly before grouping if that matches your intent

Example fix

// before
df.beam.groupby(['a', 'b'], dropna=False).sum()
// after
df = df.fillna({'a': '__missing__', 'b': '__missing__'})
df.beam.groupby(['a', 'b']).sum()

Defensive patterns

Strategy: validation

Validate before calling

if not dropna and len(group_keys) > 1:
    raise ValueError('dropna=False is unsupported for multi-column groupby in Beam')

Try / catch

try:
    g = dframe.groupby(keys).sum()
except NotImplementedError:
    g = dframe.fillna('__missing__').groupby(keys).sum()

Prevention

Fill NaN group keys before grouping instead of relying on dropna=False
Only use dropna=False with single-level grouping
Add porting checks that flag pandas dropna=False usage when migrating to Beam

When it happens

Trigger: df.beam.groupby(['a', 'b'], dropna=False) or groupby with a list of index levels producing nlevels > 1 while dropna=False

Common situations: Porting pandas code that relies on pandas' dropna=False to keep NaN group keys, with multi-column grouping

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b0cb3ab5b5304d7f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:4206

        but we only use it when necessary to avoid unnessary data transfer and
        GBKs.
    :param grouping_columns: list of column labels that were in the original
        groupby(..) ``by`` parameter. Only relevant for grouped DataFrames.
    :param grouping_indexes: list of index names (or index level numbers) to be
        grouped.
    :param kwargs: Keywords args passed to the original groupby(..) call."""
    super().__init__(expr)
    self._ungrouped = ungrouped
    self._ungrouped_with_index = ungrouped_with_index
    self._projection = projection
    self._grouping_columns = grouping_columns
    self._grouping_indexes = grouping_indexes
    self._group_keys = group_keys
    self._kwargs = kwargs

    if (self._kwargs.get('dropna', True) is False and
        self._ungrouped.proxy().index.nlevels > 1):
      raise NotImplementedError(
          "dropna=False does not work as intended in the Beam DataFrame API "
          "when grouping on multiple columns or indexes (See "
          "https://github.com/apache/beam/issues/21014).")

  def __getattr__(self, name):
    return DeferredGroupBy(
        expressions.ComputedExpression(
            'groupby_project',
            lambda gb: getattr(gb, name), [self._expr],
            requires_partition_by=partitionings.Arbitrary(),
            preserves_partition_by=partitionings.Arbitrary()),
        self._kwargs,
        self._ungrouped,
        self._ungrouped_with_index,
        self._grouping_columns,
        self._grouping_indexes,
        self._group_keys,
        projection=name)

View on GitHub (pinned to 12126d8942)