{"record":{"id":"d6e25304132ee873","repo":"apache/beam","slug":"drop-duplicates-keep-keep-r-is-not-supported-because-it-is","errorCode":null,"errorMessage":"drop_duplicates(keep={keep!r}) is not supported because it is sensitive to the order of the data. Only keep=False and keep=\"any\" are supported.","messagePattern":"drop_duplicates\\(keep=(.+?)\\) is not supported because it is sensitive to the order of the data\\. Only keep=False and keep=\"any\" are supported\\.","errorType":"exception","errorClass":"WontImplementError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/frames.py","lineNumber":2853,"sourceCode":"    return self.groupby(by).apply(\n        lambda df: pd.DataFrame(df.duplicated(keep=keep, subset=subset),\n                                columns=[None]))[None].droplevel(by)\n\n  @frame_base.with_docs_from(pd.DataFrame)\n  @frame_base.args_to_kwargs(pd.DataFrame)\n  @frame_base.populate_defaults(pd.DataFrame)\n  @frame_base.maybe_inplace\n  def drop_duplicates(self, keep, subset, ignore_index):\n    \"\"\"Only ``keep=False`` and ``keep=\"any\"`` are supported. Other values of\n    ``keep`` make this an order-sensitive operation. Note ``keep=\"any\"`` is\n    a Beam-specific option that guarantees only one duplicate will be kept, but\n    unlike ``\"first\"`` and ``\"last\"`` it makes no guarantees about _which_\n    duplicate element is kept.\"\"\"\n    # TODO(BEAM-12074): Document keep=\"any\"\n    if keep == 'any':\n      keep = 'first'\n    elif keep is not False:\n      raise frame_base.WontImplementError(\n          f\"drop_duplicates(keep={keep!r}) is not supported because it is \"\n          \"sensitive to the order of the data. Only keep=False and \"\n          \"keep=\\\"any\\\" are supported.\",\n          reason=\"order-sensitive\")\n\n    if ignore_index is not False:\n      raise frame_base.WontImplementError(\n          \"drop_duplicates(ignore_index=False) is not supported because it \"\n          \"requires generating a new index that is sensitive to the order of \"\n          \"the data.\",\n          reason=\"order-sensitive\")\n\n    by = subset or list(self.columns)\n\n    return self.groupby(by).apply(\n        lambda df: df.drop_duplicates(keep=keep, subset=subset)).droplevel(by)\n\n  @frame_base.with_docs_from(pd.DataFrame)","sourceCodeStart":2835,"sourceCodeEnd":2871,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/frames.py#L2835-L2871","documentation":"DeferredDataFrame.drop_duplicates shares the duplicated() restriction: keep='first'/'last' (including the pandas default) are order-sensitive and unsupported; only keep=False and keep='any' are allowed. Any other value raises WontImplementError.","triggerScenarios":"Calling `ddf.drop_duplicates(keep='first')`, `keep='last'`, or the default `ddf.drop_duplicates()` with no keep argument.","commonSituations":"Migrating pandas dedup code to Beam and hitting the default keep='first'; needing deterministic first-row dedup over unordered distributed data.","solutions":["Use keep='any' (Beam maps it internally) or keep=False when all duplicates should be dropped.","If 'first'/'last' semantics matter, introduce an explicit sequence/timestamp column and sort/filter within aggregations instead.","Perform the drop_duplicates on eager pandas data outside the Beam DataFrame transform."],"exampleFix":"// before\nddf.drop_duplicates()  # defaults to keep='first'\n\n// after\nddf.drop_duplicates(keep='any')","handlingStrategy":"validation","validationCode":"if keep not in ('any', False):\n    keep = 'any'","typeGuard":null,"tryCatchPattern":"from apache_beam.dataframe import frame_base\ntry:\n    out = ddf.drop_duplicates(subset=cols, keep=keep)\nexcept frame_base.WontImplementError:\n    out = ddf.drop_duplicates(subset=cols, keep='any')","preventionTips":["Always pass keep='any' or keep=False to drop_duplicates in Beam pipelines.","Never rely on the pandas default keep='first'.","For deterministic survivor choice, add a sequence column and use group-based logic."],"tags":["pandas","apache-beam","dataframe","order-sensitive","deduplication"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}