apache/beam · error · WontImplementError

append(ignore_index=True) is order sensitive because it…

Error message

append(ignore_index=True) is order sensitive because it requires generating a new index based on the order of the data.

What it means

DeferredSeries.append(ignore_index=True) is rejected because generating a fresh 0..n-1 index requires assigning positions based on the order rows arrive — exactly the order dependence Beam cannot guarantee in distributed execution (reason "order-sensitive"). Appending with ignore_index=False keeps existing labels and is allowed.

Solutions

Keep meaningful index labels and call append(to_append, ignore_index=False).
Use pd.concat([s, to_append]) and reset_index() after to_pandas() if a clean index is only needed for output.
Drop the index need entirely (e.g. write values only).

Example fix

// before
s.append(other, ignore_index=True)
// after
s.append(other, ignore_index=False)  # or: s.to_pandas().append(other, ignore_index=True)

Defensive patterns

Strategy: validation

Validate before calling

if ignore_index:
    raise ValueError("append(ignore_index=True) is order-sensitive in Beam; keep index labels or concat after to_pandas()")

Type guard

def is_index_safe(ignore_index):
    return not bool(ignore_index)

Try / catch

from apache_beam.dataframe import frame_base
try:
    combined = s.append(to_append, ignore_index=True)
except frame_base.WontImplementError:
    combined = s.to_pandas().append(to_append.to_pandas(), ignore_index=True)

Prevention

Never pass ignore_index=True to deferred append; preserve labels instead.
Treat any operation generating a fresh positional index as a Beam red flag.
Reset the index only in the final eager stage after to_pandas().

When it happens

Trigger: Calling s.append(to_append, ignore_index=True) on a DeferredSeries (with pandas < 2.0, otherwise the removal error fires first).

Common situations: Porting pandas code that resets the index after concatenation; expecting a contiguous RangeIndex in a distributed result.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/073df0f4d5ca9593. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:1385

  transpose = frame_base._elementwise_method('transpose', base=pd.Series)
  shape = property(
      frame_base.wont_implement_method(
          pd.Series, 'shape', reason="non-deferred-result"))

  @frame_base.with_docs_from(pd.Series, removed_method=PD_VERSION >= (2, 0))
  @frame_base.args_to_kwargs(pd.Series, removed_method=PD_VERSION >= (2, 0))
  @frame_base.populate_defaults(pd.Series, removed_method=PD_VERSION >= (2, 0))
  def append(self, to_append, ignore_index, verify_integrity, **kwargs):
    """``ignore_index=True`` is not supported, because it requires generating an
    order-sensitive index."""
    if PD_VERSION >= (2, 0):
      raise frame_base.WontImplementError('append() was removed in Pandas 2.0.')
    if not isinstance(to_append, DeferredSeries):
      raise frame_base.WontImplementError(
          "append() only accepts DeferredSeries instances, received " +
          str(type(to_append)))
    if ignore_index:
      raise frame_base.WontImplementError(
          "append(ignore_index=True) is order sensitive because it requires "
          "generating a new index based on the order of the data.",
          reason="order-sensitive")

    if verify_integrity:
      # We can verify the index is non-unique within index partitioned data.
      requires = partitionings.Index()
    else:
      requires = partitionings.Arbitrary()

    return frame_base.DeferredFrame.wrap(
        expressions.ComputedExpression(
            'append', lambda s, to_append: s.append(
                to_append, verify_integrity=verify_integrity, **kwargs),
            [self._expr, to_append._expr],
            requires_partition_by=requires,
            preserves_partition_by=partitionings.Arbitrary()))

View on GitHub (pinned to 12126d8942)