apache/beam · error · WontImplementError

duplicated(keep={keep!r}) is not supported because it is sen

Error message

duplicated(keep={keep!r}) is not supported because it is sensitive to the order of the data. Only keep=False and keep="any" are supported.

What it means

DeferredDataFrame.duplicated only supports keep=False or keep='any'. keep='first'/'last' require knowing which duplicate appears first in the data, which is order-sensitive in a distributed pipeline, so any other keep value raises WontImplementError.

Source

Thrown at sdks/python/apache_beam/dataframe/frames.py:2827

    return frame_base.DeferredFrame.wrap(
        expressions.ConstantExpression(pd.DataFrame.from_records(*args,
                                                                 **kwargs)))

  @frame_base.with_docs_from(pd.DataFrame)
  @frame_base.args_to_kwargs(pd.DataFrame)
  @frame_base.populate_defaults(pd.DataFrame)
  @frame_base.maybe_inplace
  def duplicated(self, keep, subset):
    """Only ``keep=False`` and ``keep="any"`` are supported. Other values of
    ``keep`` make this an order-sensitive operation. Note ``keep="any"`` is
    a Beam-specific option that guarantees only one duplicate will be kept, but
    unlike ``"first"`` and ``"last"`` it makes no guarantees about _which_
    duplicate element is kept."""
    # TODO(BEAM-12074): Document keep="any"
    if keep == 'any':
      keep = 'first'
    elif keep is not False:
      raise frame_base.WontImplementError(
          f"duplicated(keep={keep!r}) is not supported because it is "
          "sensitive to the order of the data. Only keep=False and "
          "keep=\"any\" are supported.",
          reason="order-sensitive")

    by = subset or list(self.columns)

    return self.groupby(by).apply(
        lambda df: pd.DataFrame(df.duplicated(keep=keep, subset=subset),
                                columns=[None]))[None].droplevel(by)

  @frame_base.with_docs_from(pd.DataFrame)
  @frame_base.args_to_kwargs(pd.DataFrame)
  @frame_base.populate_defaults(pd.DataFrame)
  @frame_base.maybe_inplace
  def drop_duplicates(self, keep, subset, ignore_index):
    """Only ``keep=False`` and ``keep="any"`` are supported. Other values of
    ``keep`` make this an order-sensitive operation. Note ``keep="any"`` is

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass keep=False if you want to mark all duplicates, or keep='any' if you just need exactly one survivor without order guarantees.
2. Add an explicit ordering/timestamp column and use drop-based logic if 'first' semantics are truly required.
3. Deduplicate upstream in plain pandas before creating the deferred frame.

Example fix

// before
ddf.duplicated(keep='first')

// after
ddf.duplicated(keep='any')  # or keep=False

if keep not in ('any', False):
    keep = 'any'  # Beam maps 'any' to 'first' internally

from apache_beam.dataframe import frame_base
try:
    mask = ddf.duplicated(keep=keep)
except frame_base.WontImplementError:
    mask = ddf.duplicated(keep='any')

When it happens

Trigger: Calling `ddf.duplicated(keep='first')` or `ddf.duplicated(keep='last')` (the pandas default is 'first', so even ddf.duplicated() with no args triggers this).

Common situations: Porting deduplication logic that relies on first-occurrence semantics; forgetting that the pandas default keep='first' is unsupported in Beam.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• drop_duplicates(keep={keep!r}) is not supported because it i
• fillna(method={method!r}, axis={axis!r}) is not supported be
• fillna(limit={method!r}, axis={axis!r}) is not supported bec
• Grouping by a concrete ndarray is order sensitive.
• replace(method={method!r}) is not supported because it is or

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/43c051a1641785e3. Report an issue: GitHub.