{"record":{"id":"43c051a1641785e3","repo":"apache/beam","slug":"duplicated-keep-keep-r-is-not-supported-because-it-is","errorCode":null,"errorMessage":"duplicated(keep={keep!r}) is not supported because it is sensitive to the order of the data. Only keep=False and keep=\"any\" are supported.","messagePattern":"duplicated\\(keep=(.+?)\\) is not supported because it is sensitive to the order of the data\\. Only keep=False and keep=\"any\" are supported\\.","errorType":"exception","errorClass":"WontImplementError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/frames.py","lineNumber":2827,"sourceCode":"    return frame_base.DeferredFrame.wrap(\n        expressions.ConstantExpression(pd.DataFrame.from_records(*args,\n                                                                 **kwargs)))\n\n  @frame_base.with_docs_from(pd.DataFrame)\n  @frame_base.args_to_kwargs(pd.DataFrame)\n  @frame_base.populate_defaults(pd.DataFrame)\n  @frame_base.maybe_inplace\n  def duplicated(self, keep, subset):\n    \"\"\"Only ``keep=False`` and ``keep=\"any\"`` are supported. Other values of\n    ``keep`` make this an order-sensitive operation. Note ``keep=\"any\"`` is\n    a Beam-specific option that guarantees only one duplicate will be kept, but\n    unlike ``\"first\"`` and ``\"last\"`` it makes no guarantees about _which_\n    duplicate element is kept.\"\"\"\n    # TODO(BEAM-12074): Document keep=\"any\"\n    if keep == 'any':\n      keep = 'first'\n    elif keep is not False:\n      raise frame_base.WontImplementError(\n          f\"duplicated(keep={keep!r}) is not supported because it is \"\n          \"sensitive to the order of the data. Only keep=False and \"\n          \"keep=\\\"any\\\" are supported.\",\n          reason=\"order-sensitive\")\n\n    by = subset or list(self.columns)\n\n    return self.groupby(by).apply(\n        lambda df: pd.DataFrame(df.duplicated(keep=keep, subset=subset),\n                                columns=[None]))[None].droplevel(by)\n\n  @frame_base.with_docs_from(pd.DataFrame)\n  @frame_base.args_to_kwargs(pd.DataFrame)\n  @frame_base.populate_defaults(pd.DataFrame)\n  @frame_base.maybe_inplace\n  def drop_duplicates(self, keep, subset, ignore_index):\n    \"\"\"Only ``keep=False`` and ``keep=\"any\"`` are supported. Other values of\n    ``keep`` make this an order-sensitive operation. Note ``keep=\"any\"`` is","sourceCodeStart":2809,"sourceCodeEnd":2845,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/frames.py#L2809-L2845","documentation":"DeferredDataFrame.duplicated only supports keep=False or keep='any'. keep='first'/'last' require knowing which duplicate appears first in the data, which is order-sensitive in a distributed pipeline, so any other keep value raises WontImplementError.","triggerScenarios":"Calling `ddf.duplicated(keep='first')` or `ddf.duplicated(keep='last')` (the pandas default is 'first', so even ddf.duplicated() with no args triggers this).","commonSituations":"Porting deduplication logic that relies on first-occurrence semantics; forgetting that the pandas default keep='first' is unsupported in Beam.","solutions":["Pass keep=False if you want to mark all duplicates, or keep='any' if you just need exactly one survivor without order guarantees.","Add an explicit ordering/timestamp column and use drop-based logic if 'first' semantics are truly required.","Deduplicate upstream in plain pandas before creating the deferred frame."],"exampleFix":"// before\nddf.duplicated(keep='first')\n\n// after\nddf.duplicated(keep='any')  # or keep=False","handlingStrategy":"validation","validationCode":"if keep not in ('any', False):\n    keep = 'any'  # Beam maps 'any' to 'first' internally","typeGuard":null,"tryCatchPattern":"from apache_beam.dataframe import frame_base\ntry:\n    mask = ddf.duplicated(keep=keep)\nexcept frame_base.WontImplementError:\n    mask = ddf.duplicated(keep='any')","preventionTips":["Remember pandas' default keep='first' is invalid in Beam — always pass keep explicitly.","Use keep='any' unless true first-occurrence semantics are required.","Add an explicit ordering column if you need deterministic dedup."],"tags":["pandas","apache-beam","dataframe","order-sensitive","deduplication"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}