pathwaycom/pathway · error · ValueError

index of resulting DataFrame must be unique

Error message

index of resulting DataFrame must be unique

What it means

When no output_universe is designated, the pandas adapter treats the returned DataFrame's index as the output universe and (if it does not already contain Pathway Pointers) will map it to references. That mapping requires unique index values, so a duplicated index raises ValueError('index of resulting DataFrame must be unique') from process_pandas_output.

Source

Thrown at python/pathway/stdlib/utils/pandas_transformer.py:78

    func = func_spec.func

    def process_pandas_output(
        result: pd.DataFrame | pd.Series, pandas_input: list[pd.DataFrame] = []
    ):
        if isinstance(result, pd.Series):
            result = pd.DataFrame(result)

        result.columns = output_schema.column_names()  # type: ignore

        if output_universe_arg_index is not None and not result.index.equals(
            pandas_input[output_universe_arg_index].index
        ):
            raise ValueError(
                "resulting universe does not match the universe of the indicated argument"
            )
        else:
            if not result.index.is_unique:
                raise ValueError("index of resulting DataFrame must be unique")
            index_as_series = result.index.to_series()
            if not index_as_series.map(lambda x: isinstance(x, Pointer)).all():
                new_index = index_as_series.map(lambda x: ref_scalar(x))
                result.reindex(new_index)
            assert result.index.is_unique

        return result

    if len(func_spec.arg_names) == 0:
        result = func()
        result = process_pandas_output(result)
        output = table_from_pandas(result)
    else:
        input_table = _create_input_table(*inputs)

        def wrapper(*input_tables, inputs=inputs):
            pandas_input = []
            for idx, table in enumerate(input_tables):

View on GitHub (pinned to fa2f74a464)

Solutions

1. Designate an output_universe argument (name or index) so the returned index must equal a designated input's index instead of being deduped implicitly
2. Deduplicate or regenerate a unique index before returning, e.g. result = result.set_index(pd.RangeIndex(len(result)))
3. If one row should stay one row, make the UDF return exactly one row per input row indexed by the input index

Example fix

# before
@pw.pandas_transformer()
def f(df: pd.DataFrame) -> pd.DataFrame:
    return df.explode('tags')  # duplicated index

# after
@pw.pandas_transformer(output_universe='df')
def f(df: pd.DataFrame) -> pd.DataFrame:
    r = df.explode('tags')
    return r  # output_universe mode requires preserving df.index; else:
    # r = r.reset_index(drop=True)  # unique RangeIndex when no output_universe

# inside the UDF, before returning:
assert result.index.is_unique, 'return a unique index or set output_universe'

try:
    table = adapter_call(udf_result)
except ValueError as e:
    if 'index of resulting DataFrame must be unique' in str(e):
        result = result.reset_index(drop=True)  # or set output_universe and retry
    else:
        raise

When it happens

Trigger: The UDF returns a frame after a Cartesian/join explosion, groupby result without a unique key, or concat producing repeated labels; returning a Series converted to a DataFrame where the index repeats; head/tail operations leaving duplicated labels from a prior merge.

Common situations: One-to-many enrichments inside pandas UDFs (one input row producing several output rows) without designating an output_universe; UDFs adapted from notebook code that ignores index uniqueness; results from pd.concat of overlapping frames.

Related errors

• resulting universe does not match the universe of the indica
• wrong output universe. No argument of name: {arg}
• wrong output universe. Index out of range
• string {freq} cannot be parsed as a duration
• You cannot use {dep.to_column_expression()} in this context.

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/a97d826c1e6c88a1. Report an issue: GitHub.