{"record":{"id":"a97d826c1e6c88a1","repo":"pathwaycom/pathway","slug":"index-of-resulting-dataframe-must-be-unique","errorCode":null,"errorMessage":"index of resulting DataFrame must be unique","messagePattern":"index of resulting DataFrame must be unique","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/stdlib/utils/pandas_transformer.py","lineNumber":78,"sourceCode":"    func = func_spec.func\n\n    def process_pandas_output(\n        result: pd.DataFrame | pd.Series, pandas_input: list[pd.DataFrame] = []\n    ):\n        if isinstance(result, pd.Series):\n            result = pd.DataFrame(result)\n\n        result.columns = output_schema.column_names()  # type: ignore\n\n        if output_universe_arg_index is not None and not result.index.equals(\n            pandas_input[output_universe_arg_index].index\n        ):\n            raise ValueError(\n                \"resulting universe does not match the universe of the indicated argument\"\n            )\n        else:\n            if not result.index.is_unique:\n                raise ValueError(\"index of resulting DataFrame must be unique\")\n            index_as_series = result.index.to_series()\n            if not index_as_series.map(lambda x: isinstance(x, Pointer)).all():\n                new_index = index_as_series.map(lambda x: ref_scalar(x))\n                result.reindex(new_index)\n            assert result.index.is_unique\n\n        return result\n\n    if len(func_spec.arg_names) == 0:\n        result = func()\n        result = process_pandas_output(result)\n        output = table_from_pandas(result)\n    else:\n        input_table = _create_input_table(*inputs)\n\n        def wrapper(*input_tables, inputs=inputs):\n            pandas_input = []\n            for idx, table in enumerate(input_tables):","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/stdlib/utils/pandas_transformer.py#L60-L96","documentation":"When no output_universe is designated, the pandas adapter treats the returned DataFrame's index as the output universe and (if it does not already contain Pathway Pointers) will map it to references. That mapping requires unique index values, so a duplicated index raises ValueError('index of resulting DataFrame must be unique') from process_pandas_output.","triggerScenarios":"The UDF returns a frame after a Cartesian/join explosion, groupby result without a unique key, or concat producing repeated labels; returning a Series converted to a DataFrame where the index repeats; head/tail operations leaving duplicated labels from a prior merge.","commonSituations":"One-to-many enrichments inside pandas UDFs (one input row producing several output rows) without designating an output_universe; UDFs adapted from notebook code that ignores index uniqueness; results from pd.concat of overlapping frames.","solutions":["Designate an output_universe argument (name or index) so the returned index must equal a designated input's index instead of being deduped implicitly","Deduplicate or regenerate a unique index before returning, e.g. result = result.set_index(pd.RangeIndex(len(result)))","If one row should stay one row, make the UDF return exactly one row per input row indexed by the input index"],"exampleFix":"# before\n@pw.pandas_transformer()\ndef f(df: pd.DataFrame) -> pd.DataFrame:\n    return df.explode('tags')  # duplicated index\n\n# after\n@pw.pandas_transformer(output_universe='df')\ndef f(df: pd.DataFrame) -> pd.DataFrame:\n    r = df.explode('tags')\n    return r  # output_universe mode requires preserving df.index; else:\n    # r = r.reset_index(drop=True)  # unique RangeIndex when no output_universe","handlingStrategy":"validation","validationCode":"# inside the UDF, before returning:\nassert result.index.is_unique, 'return a unique index or set output_universe'","typeGuard":null,"tryCatchPattern":"try:\n    table = adapter_call(udf_result)\nexcept ValueError as e:\n    if 'index of resulting DataFrame must be unique' in str(e):\n        result = result.reset_index(drop=True)  # or set output_universe and retry\n    else:\n        raise","preventionTips":["Set output_universe whenever the UDF is conceptively row-per-row","After explode/merge/concat, regenerate a unique index with reset_index(drop=True)","Unit-test the UDF's index contract (unique, or equal to the designated input's index)"],"tags":["pathway","pandas","udf","universe","duplicate-index"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}