apache/beam · error · WontImplementError

drop_duplicates(ignore_index=False) is not supported…

Error message

drop_duplicates(ignore_index=False) is not supported because it requires generating a new index that is sensitive to the order of the data.

What it means

DeferredDataFrame.drop_duplicates raises when ignore_index is not False, because resetting to a fresh RangeIndex requires numbering rows by their order in the data — order-sensitive in a distributed pipeline.

Solutions

Omit ignore_index or set it to False; keep the existing index.
Reset the index after collecting results to the driver: result.to_pandas().reset_index(drop=True).
Drop the index column entirely if downstream code doesn't need it.

Example fix

// before
ddf.drop_duplicates(subset='id', ignore_index=True)

// after
ddf.drop_duplicates(subset='id')  # reset index later on the eager result

Defensive patterns

Strategy: validation

Validate before calling

if ignore_index is not False:
    ignore_index = False  # reset the index later on the eager result instead

Try / catch

from apache_beam.dataframe import frame_base
try:
    out = ddf.drop_duplicates(ignore_index=ignore_index)
except frame_base.WontImplementError:
    out = ddf.drop_duplicates()  # then result.to_pandas().reset_index(drop=True)

Prevention

Never pass ignore_index=True to deferred frames.
Reset indexes after collecting results to the driver with to_pandas().
Treat the deferred index as unordered; don't rely on it downstream.

When it happens

Trigger: Calling `ddf.drop_duplicates(..., ignore_index=True)` (or any non-False value) on a DeferredDataFrame.

Common situations: Porting pandas >= 1.0 code that uses ignore_index=True for convenience after dedup, without realizing the resulting index is meaningless in Beam.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d52b3713dda7a58a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:2860

  @frame_base.maybe_inplace
  def drop_duplicates(self, keep, subset, ignore_index):
    """Only ``keep=False`` and ``keep="any"`` are supported. Other values of
    ``keep`` make this an order-sensitive operation. Note ``keep="any"`` is
    a Beam-specific option that guarantees only one duplicate will be kept, but
    unlike ``"first"`` and ``"last"`` it makes no guarantees about _which_
    duplicate element is kept."""
    # TODO(BEAM-12074): Document keep="any"
    if keep == 'any':
      keep = 'first'
    elif keep is not False:
      raise frame_base.WontImplementError(
          f"drop_duplicates(keep={keep!r}) is not supported because it is "
          "sensitive to the order of the data. Only keep=False and "
          "keep=\"any\" are supported.",
          reason="order-sensitive")

    if ignore_index is not False:
      raise frame_base.WontImplementError(
          "drop_duplicates(ignore_index=False) is not supported because it "
          "requires generating a new index that is sensitive to the order of "
          "the data.",
          reason="order-sensitive")

    by = subset or list(self.columns)

    return self.groupby(by).apply(
        lambda df: df.drop_duplicates(keep=keep, subset=subset)).droplevel(by)

  @frame_base.with_docs_from(pd.DataFrame)
  @frame_base.args_to_kwargs(pd.DataFrame)
  @frame_base.populate_defaults(pd.DataFrame)
  def aggregate(self, func, axis, *args, **kwargs):
    # We have specialized implementations for these.
    if func in ('quantile',):
      return getattr(self, func)(*args, axis=axis, **kwargs)

View on GitHub (pinned to 12126d8942)