apache/beam · error · WontImplementError

drop_duplicates(keep={keep!r}) is not supported because it i

Error message

drop_duplicates(keep={keep!r}) is not supported because it is sensitive to the order of the data. Only keep=False and keep="any" are supported.

What it means

DeferredDataFrame.drop_duplicates shares the duplicated() restriction: keep='first'/'last' (including the pandas default) are order-sensitive and unsupported; only keep=False and keep='any' are allowed. Any other value raises WontImplementError.

Source

Thrown at sdks/python/apache_beam/dataframe/frames.py:2853

    return self.groupby(by).apply(
        lambda df: pd.DataFrame(df.duplicated(keep=keep, subset=subset),
                                columns=[None]))[None].droplevel(by)

  @frame_base.with_docs_from(pd.DataFrame)
  @frame_base.args_to_kwargs(pd.DataFrame)
  @frame_base.populate_defaults(pd.DataFrame)
  @frame_base.maybe_inplace
  def drop_duplicates(self, keep, subset, ignore_index):
    """Only ``keep=False`` and ``keep="any"`` are supported. Other values of
    ``keep`` make this an order-sensitive operation. Note ``keep="any"`` is
    a Beam-specific option that guarantees only one duplicate will be kept, but
    unlike ``"first"`` and ``"last"`` it makes no guarantees about _which_
    duplicate element is kept."""
    # TODO(BEAM-12074): Document keep="any"
    if keep == 'any':
      keep = 'first'
    elif keep is not False:
      raise frame_base.WontImplementError(
          f"drop_duplicates(keep={keep!r}) is not supported because it is "
          "sensitive to the order of the data. Only keep=False and "
          "keep=\"any\" are supported.",
          reason="order-sensitive")

    if ignore_index is not False:
      raise frame_base.WontImplementError(
          "drop_duplicates(ignore_index=False) is not supported because it "
          "requires generating a new index that is sensitive to the order of "
          "the data.",
          reason="order-sensitive")

    by = subset or list(self.columns)

    return self.groupby(by).apply(
        lambda df: df.drop_duplicates(keep=keep, subset=subset)).droplevel(by)

  @frame_base.with_docs_from(pd.DataFrame)

View on GitHub (pinned to 12126d8942)

Solutions

Use keep='any' (Beam maps it internally) or keep=False when all duplicates should be dropped.
If 'first'/'last' semantics matter, introduce an explicit sequence/timestamp column and sort/filter within aggregations instead.
Perform the drop_duplicates on eager pandas data outside the Beam DataFrame transform.

Example fix

// before
ddf.drop_duplicates()  # defaults to keep='first'

// after
ddf.drop_duplicates(keep='any')

Defensive patterns

Strategy: validation

Validate before calling

if keep not in ('any', False):
    keep = 'any'

Try / catch

from apache_beam.dataframe import frame_base
try:
    out = ddf.drop_duplicates(subset=cols, keep=keep)
except frame_base.WontImplementError:
    out = ddf.drop_duplicates(subset=cols, keep='any')

Prevention

Always pass keep='any' or keep=False to drop_duplicates in Beam pipelines.
Never rely on the pandas default keep='first'.
For deterministic survivor choice, add a sequence column and use group-based logic.

When it happens

Trigger: Calling `ddf.drop_duplicates(keep='first')`, `keep='last'`, or the default `ddf.drop_duplicates()` with no keep argument.

Common situations: Migrating pandas dedup code to Beam and hitting the default keep='first'; needing deterministic first-row dedup over unordered distributed data.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d6e25304132ee873. Report an issue: GitHub.