apache/beam · error · NotImplementedError

set_index with Index or Series instances is not yet supporte

Error message

set_index with Index or Series instances is not yet supported (https://github.com/apache/beam/issues/20759).

What it means

DeferredDataFrame.set_index supports only column names (strings) as keys. Passing a pandas Index/Series-like value as a DeferredIndex or DeferredFrame key is not yet implemented (tracked in apache/beam issue #20759) because it is generally order-sensitive in a distributed setting, so a NotImplementedError is raised.

Source

Thrown at sdks/python/apache_beam/dataframe/frames.py:2677

    if key in self.columns:
      return self[key]
    else:
      return default_value

  @frame_base.with_docs_from(pd.DataFrame)
  @frame_base.args_to_kwargs(pd.DataFrame)
  @frame_base.populate_defaults(pd.DataFrame)
  @frame_base.maybe_inplace
  def set_index(self, keys, **kwargs):
    """``keys`` must be a ``str`` or ``list[str]``. Passing an Index or Series
    is not yet supported (`Issue 20759
    <https://github.com/apache/beam/issues/20759>`_)."""
    if isinstance(keys, str):
      keys = [keys]

    if any(isinstance(k, (_DeferredIndex, frame_base.DeferredFrame))
           for k in keys):
      raise NotImplementedError("set_index with Index or Series instances is "
                                "not yet supported "
                                "(https://github.com/apache/beam/issues/20759)"
                                ".")

    return frame_base.DeferredFrame.wrap(
      expressions.ComputedExpression(
          'set_index',
          lambda df: df.set_index(keys, **kwargs),
          [self._expr],
          requires_partition_by=partitionings.Arbitrary(),
          preserves_partition_by=partitionings.Singleton()))


  @frame_base.with_docs_from(pd.DataFrame)
  @frame_base.args_to_kwargs(pd.DataFrame)
  @frame_base.populate_defaults(
      pd.DataFrame,
      removed_args=['inplace'] if PD_VERSION >= (2, 0) else None)

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass the column name string instead of the Series: df.set_index('col') rather than df.set_index(df['col']).
2. Pass a list of column-name strings for a MultiIndex: df.set_index(['a', 'b']).
3. Materialize the desired index into a column first (df['idx'] = ...), then set_index('idx').
4. Convert to plain pandas for this step if a Series-based index is unavoidable.

Example fix

// before
df = df.set_index(df['user_id'])
// after
df = df.set_index('user_id')

if isinstance(keys, (DeferredIndex, DeferredFrame)) or any(isinstance(k, (DeferredIndex, DeferredFrame)) for k in (keys if isinstance(keys, list) else [keys])):
    raise TypeError('use column name strings with set_index')

def is_valid_set_index_keys(keys) -> bool:
    keys = [keys] if isinstance(keys, str) else keys
    return all(isinstance(k, str) and not isinstance(k, (DeferredIndex, DeferredFrame)) for k in keys)

try:
    df = df.set_index(keys)
except NotImplementedError:
    df = df.set_index('col_name')

When it happens

Trigger: df.set_index(df['col']) (passing a deferred Series instead of the name 'col'); df.set_index(pd.Index([...])) or df.set_index(other_df.index).

Common situations: Porting pandas code that builds indexes from Series objects; dynamically constructed indexes in pipelines migrated to apache_beam.dataframe.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• groupby(as_index=False)
• by
• Assigning an index is not yet supported. Consider using set_
• per-level align
• Indexing with a non-bool deferred frame is not yet supported

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d3a1de444f030e2a. Report an issue: GitHub.