{"record":{"id":"d52b3713dda7a58a","repo":"apache/beam","slug":"drop-duplicates-ignore-index-false-is-not-supported-because","errorCode":null,"errorMessage":"drop_duplicates(ignore_index=False) is not supported because it requires generating a new index that is sensitive to the order of the data.","messagePattern":"drop_duplicates\\(ignore_index=False\\) is not supported because it requires generating a new index that is sensitive to the order of the data\\.","errorType":"exception","errorClass":"WontImplementError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/frames.py","lineNumber":2860,"sourceCode":"  @frame_base.maybe_inplace\n  def drop_duplicates(self, keep, subset, ignore_index):\n    \"\"\"Only ``keep=False`` and ``keep=\"any\"`` are supported. Other values of\n    ``keep`` make this an order-sensitive operation. Note ``keep=\"any\"`` is\n    a Beam-specific option that guarantees only one duplicate will be kept, but\n    unlike ``\"first\"`` and ``\"last\"`` it makes no guarantees about _which_\n    duplicate element is kept.\"\"\"\n    # TODO(BEAM-12074): Document keep=\"any\"\n    if keep == 'any':\n      keep = 'first'\n    elif keep is not False:\n      raise frame_base.WontImplementError(\n          f\"drop_duplicates(keep={keep!r}) is not supported because it is \"\n          \"sensitive to the order of the data. Only keep=False and \"\n          \"keep=\\\"any\\\" are supported.\",\n          reason=\"order-sensitive\")\n\n    if ignore_index is not False:\n      raise frame_base.WontImplementError(\n          \"drop_duplicates(ignore_index=False) is not supported because it \"\n          \"requires generating a new index that is sensitive to the order of \"\n          \"the data.\",\n          reason=\"order-sensitive\")\n\n    by = subset or list(self.columns)\n\n    return self.groupby(by).apply(\n        lambda df: df.drop_duplicates(keep=keep, subset=subset)).droplevel(by)\n\n  @frame_base.with_docs_from(pd.DataFrame)\n  @frame_base.args_to_kwargs(pd.DataFrame)\n  @frame_base.populate_defaults(pd.DataFrame)\n  def aggregate(self, func, axis, *args, **kwargs):\n    # We have specialized implementations for these.\n    if func in ('quantile',):\n      return getattr(self, func)(*args, axis=axis, **kwargs)\n","sourceCodeStart":2842,"sourceCodeEnd":2878,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/frames.py#L2842-L2878","documentation":"DeferredDataFrame.drop_duplicates raises when ignore_index is not False, because resetting to a fresh RangeIndex requires numbering rows by their order in the data — order-sensitive in a distributed pipeline.","triggerScenarios":"Calling `ddf.drop_duplicates(..., ignore_index=True)` (or any non-False value) on a DeferredDataFrame.","commonSituations":"Porting pandas >= 1.0 code that uses ignore_index=True for convenience after dedup, without realizing the resulting index is meaningless in Beam.","solutions":["Omit ignore_index or set it to False; keep the existing index.","Reset the index after collecting results to the driver: result.to_pandas().reset_index(drop=True).","Drop the index column entirely if downstream code doesn't need it."],"exampleFix":"// before\nddf.drop_duplicates(subset='id', ignore_index=True)\n\n// after\nddf.drop_duplicates(subset='id')  # reset index later on the eager result","handlingStrategy":"validation","validationCode":"if ignore_index is not False:\n    ignore_index = False  # reset the index later on the eager result instead","typeGuard":null,"tryCatchPattern":"from apache_beam.dataframe import frame_base\ntry:\n    out = ddf.drop_duplicates(ignore_index=ignore_index)\nexcept frame_base.WontImplementError:\n    out = ddf.drop_duplicates()  # then result.to_pandas().reset_index(drop=True)","preventionTips":["Never pass ignore_index=True to deferred frames.","Reset indexes after collecting results to the driver with to_pandas().","Treat the deferred index as unordered; don't rely on it downstream."],"tags":["pandas","apache-beam","dataframe","order-sensitive"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}