{"record":{"id":"b0cb3ab5b5304d7f","repo":"apache/beam","slug":"dropna-false-does-not-work-as-intended-in-the-beam-dataframe","errorCode":null,"errorMessage":"dropna=False does not work as intended in the Beam DataFrame API when grouping on multiple columns or indexes (See https://github.com/apache/beam/issues/21014).","messagePattern":"dropna=False does not work as intended in the Beam DataFrame API when grouping on multiple columns or indexes \\(See https://github\\.com/apache/beam/issues/21014\\)\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/frames.py","lineNumber":4206,"sourceCode":"        but we only use it when necessary to avoid unnessary data transfer and\n        GBKs.\n    :param grouping_columns: list of column labels that were in the original\n        groupby(..) ``by`` parameter. Only relevant for grouped DataFrames.\n    :param grouping_indexes: list of index names (or index level numbers) to be\n        grouped.\n    :param kwargs: Keywords args passed to the original groupby(..) call.\"\"\"\n    super().__init__(expr)\n    self._ungrouped = ungrouped\n    self._ungrouped_with_index = ungrouped_with_index\n    self._projection = projection\n    self._grouping_columns = grouping_columns\n    self._grouping_indexes = grouping_indexes\n    self._group_keys = group_keys\n    self._kwargs = kwargs\n\n    if (self._kwargs.get('dropna', True) is False and\n        self._ungrouped.proxy().index.nlevels > 1):\n      raise NotImplementedError(\n          \"dropna=False does not work as intended in the Beam DataFrame API \"\n          \"when grouping on multiple columns or indexes (See \"\n          \"https://github.com/apache/beam/issues/21014).\")\n\n  def __getattr__(self, name):\n    return DeferredGroupBy(\n        expressions.ComputedExpression(\n            'groupby_project',\n            lambda gb: getattr(gb, name), [self._expr],\n            requires_partition_by=partitionings.Arbitrary(),\n            preserves_partition_by=partitionings.Arbitrary()),\n        self._kwargs,\n        self._ungrouped,\n        self._ungrouped_with_index,\n        self._grouping_columns,\n        self._grouping_indexes,\n        self._group_keys,\n        projection=name)","sourceCodeStart":4188,"sourceCodeEnd":4224,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/frames.py#L4188-L4224","documentation":"DeferredGroupBy raises NotImplementedError when dropna=False is combined with grouping on multiple columns/indexes (index.nlevels > 1), because Beam's grouping cannot keep NaN-key groups correctly in that case; see GitHub issue 21014.","triggerScenarios":"df.beam.groupby(['a', 'b'], dropna=False) or groupby with a list of index levels producing nlevels > 1 while dropna=False","commonSituations":"Porting pandas code that relies on pandas' dropna=False to keep NaN group keys, with multi-column grouping","solutions":["Fill NaN values in the grouping columns before grouping (e.g. fillna('__missing__')) and keep dropna default True","Group on a single column/index level when using dropna=False","Drop NaN-key rows explicitly before grouping if that matches your intent"],"exampleFix":"// before\ndf.beam.groupby(['a', 'b'], dropna=False).sum()\n// after\ndf = df.fillna({'a': '__missing__', 'b': '__missing__'})\ndf.beam.groupby(['a', 'b']).sum()","handlingStrategy":"validation","validationCode":"if not dropna and len(group_keys) > 1:\n    raise ValueError('dropna=False is unsupported for multi-column groupby in Beam')","typeGuard":null,"tryCatchPattern":"try:\n    g = dframe.groupby(keys).sum()\nexcept NotImplementedError:\n    g = dframe.fillna('__missing__').groupby(keys).sum()","preventionTips":["Fill NaN group keys before grouping instead of relying on dropna=False","Only use dropna=False with single-level grouping","Add porting checks that flag pandas dropna=False usage when migrating to Beam"],"tags":["python","apache-beam","dataframe","groupby","not-implemented"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}