pandas-dev/pandas · error · NotImplementedError

The index/columns must be unique when raw=False and engine='

Error message

The index/columns must be unique when raw=False and engine='numba'

What it means

Raised by apply_series_numba (apply.py:1310-1313) when engine='numba' is used with raw=False on a frame whose index or columns are not unique. The numba Series-passing path assembles results by positionally re-attaching them to the existing index/columns, which is only unambiguous when labels are unique; duplicate labels would make the result ambiguous, so pandas requires uniqueness up front.

Source

Thrown at pandas/core/apply.py:1311

        results = {}

        for i, v in enumerate(series_gen):
            results[i] = self.func(v, *self.args, **self.kwargs)
            if isinstance(results[i], ABCSeries):
                # If we have a view on v, we need to make a copy because
                #  series_generator will swap out the underlying data
                results[i] = results[i].copy(deep=False)

        return results, res_index

    def apply_series_numba(self):
        if self.engine_kwargs.get("parallel", False):
            raise NotImplementedError(
                "Parallel apply is not supported when raw=False and engine='numba'"
            )
        if not self.obj.index.is_unique or not self.columns.is_unique:
            raise NotImplementedError(
                "The index/columns must be unique when raw=False and engine='numba'"
            )
        self.validate_values_for_numba()
        results = self.apply_with_numba()
        return results, self.result_index

    def wrap_results(self, results: ResType, res_index: Index) -> DataFrame | Series:
        from pandas import Series

        # see if we can infer the results
        if len(results) > 0 and 0 in results and is_sequence(results[0]):
            return self.wrap_results_for_axis(results, res_index)

        # dict of scalars

        # the default dtype of an empty Series is `object`, but this
        # code can be hit by df.mean() where the result should have dtype
        # float64 even if it's an empty Series.

View on GitHub (pinned to 71959b8cb9)

Solutions

1. Reset or deduplicate the index before applying: df = df.reset_index(drop=True).
2. Deduplicate columns by renaming or dropping dupes: df.columns = pd.io.parsers ParserBase... or df.loc[:, ~df.columns.duplicated()].
3. Switch to raw=True (the raw numba path does not require label uniqueness).
4. Fall back to engine='python' if label uniqueness must be preserved.

Example fix

// before
df.apply(func, engine='numba')  # df has duplicate index values
// after
df = df.reset_index(drop=True)
df.apply(func, engine='numba')

if engine == 'numba' and raw is False:
    if not df.index.is_unique:
        raise ValueError('numba engine requires a unique index; call reset_index(drop=True)')
    if hasattr(df, 'columns') and not df.columns.is_unique:
        raise ValueError('numba engine requires unique column names')

def safe_for_numba_series_apply(df, engine: str, raw: bool) -> bool:
    if engine != 'numba' or raw:
        return True
    idx_ok = getattr(df, 'index', None) is None or df.index.is_unique
    col_ok = not hasattr(df, 'columns') or df.columns.is_unique
    return idx_ok and col_ok

try:
    df.apply(func, engine='numba')
except NotImplementedError as e:
    if 'index/columns must be unique' in str(e).lower():
        df.reset_index(drop=True).apply(func, engine='numba')
    else:
        raise

When it happens

Trigger: df.apply(func, engine='numba') (raw defaults to False) on a DataFrame with duplicate index labels or duplicate column names. Triggered at apply.py:1310-1313 when self.obj.index.is_unique or self.columns.is_unique is False.

Common situations: Frames built from concat/merge/join operations that retained duplicate labels; CSVs whose key column has dupes used as the index; intentionally duplicated columns from a pivot or concatenation step; transitioning a workflow to engine='numba' without de-duplicating labels.

Related errors

• The 'numba' engine doesn't support list-like/dict likes of c
• the 'numba' engine doesn't support using a numpy ufunc as th
• the 'numba' engine doesn't support result_type='broadcast'
• Parallel apply is not supported when raw=False and engine='n
• The numba engine only supports using string or numeric colum

AI-assisted analysis of pandas-dev/pandas@71959b8cb9 (2026-08-07). Data as JSON: /api/errors/c5d974aad93ec3c0. Report an issue: GitHub.