apache/beam · error · NotImplementedError

concat(ignore_index)

Error message

concat(ignore_index)

What it means

The Beam DataFrame API's pd.concat wrapper does not support ignore_index=True, because renumbering indexes across concatenated frames conflicts with its distributed index-preserving semantics. Passing ignore_index=True raises NotImplementedError early.

Source

Thrown at sdks/python/apache_beam/dataframe/pandas_top_level_functions.py:100

  bdate_range = _defer_to_pandas('bdate_range')

  @staticmethod
  @frame_base.args_to_kwargs(pd)
  @frame_base.populate_defaults(pd)
  def concat(
      objs,
      axis,
      join,
      ignore_index,
      keys,
      levels,
      names,
      verify_integrity,
      sort,
      copy):

    if ignore_index:
      raise NotImplementedError('concat(ignore_index)')
    if levels:
      raise NotImplementedError('concat(levels)')

    if isinstance(objs, Mapping):
      if keys is None:
        keys = list(objs.keys())
      objs = [objs[k] for k in keys]
    else:
      objs = list(objs)

    if keys is None:
      preserves_partitioning = partitionings.Arbitrary()
    else:
      # Index 0 will be a new index for keys, only partitioning by the original
      # indexes (1 to N) will be preserved.
      nlevels = min(o._expr.proxy().index.nlevels for o in objs)
      preserves_partitioning = partitionings.Index(
          [i for i in range(1, nlevels + 1)])

View on GitHub (pinned to 12126d8942)

Solutions

1. Use ignore_index=False (the default) and call reset_index() on the result if a fresh index is needed
2. Manually drop/rebuild the index after concat: df.reset_index(drop=True) on the deferred result
3. Restructure the pipeline so frames are concatenated outside Beam (in regular pandas) if index reset semantics are essential

Example fix

// before
pd.concat([df1, df2], ignore_index=True)
// after
pd.concat([df1, df2]).reset_index(drop=True)

if ignore_index:
    ignore_index = False  # not supported; reset index after concat instead

try:
    out = pd.concat(objs, ignore_index=ignore_index)
except NotImplementedError:
    out = pd.concat(objs).reset_index(drop=True)

When it happens

Trigger: Calling pd.concat(..., ignore_index=True) on deferred Beam dataframes within a Beam DataFrame transform.

Common situations: Porting pandas code that resets the index on concat; combining frames with duplicate index values and expecting a fresh default index.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• concat(levels)
• groupby(as_index=False)
• by
• Assigning an index is not yet supported. Consider using set_
• per-level align

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5077d619356c5876. Report an issue: GitHub.