apache/beam · error · WontImplementError

compare(align_axis= , keep_shape= ) is not allowed because…

Error message

compare(align_axis={align_axis!r}, keep_shape={keep_shape!r}) is not allowed because the output columns depend on the data, please specify keep_shape=True.

What it means

apache_beam.dataframe raises WontImplementError for DataFrame.compare() when align_axis is 1/'columns' and keep_shape is not True. In Beam's deferred model, the output columns would be determined by comparing the actual data at execution time, which the pipeline cannot do ahead of time. The library forbids data-dependent columns and asks you to pass keep_shape=True so the column set is fixed.

Solutions

Pass keep_shape=True: df.compare(other, align_axis='columns', keep_shape=True).
Use the default align_axis=0 ('rows') so the comparison produces rows instead of data-dependent columns.
Restructure the comparison manually (e.g. select and rename columns yourself) so the output schema is static.
If a data-dependent column layout is truly required, run that step in plain pandas outside the Beam pipeline.

Example fix

// before
diff = df.compare(other_df)

// after
diff = df.compare(other_df, align_axis='columns', keep_shape=True)

Defensive patterns

Strategy: validation

Validate before calling

if align_axis in (1, 'columns') and not keep_shape:
    raise ValueError("Use keep_shape=True with align_axis=1 in Beam dataframes")

Try / catch

try:
    diff = df.compare(other, align_axis='columns')
except apachebeam.WontImplementError:
    diff = df.compare(other, align_axis='columns', keep_shape=True)

Prevention

Never call compare() with align_axis=1/'columns' on deferred frames without keep_shape=True.
Prefer align_axis=0 (row-wise diffs) in Beam pipelines.
Review ported pandas code for column-creating operations before moving them into Beam.

When it happens

Trigger: Calling df.compare(other) or df.compare(other, align_axis=1) / align_axis='columns' without keep_shape=True on a deferred Beam DataFrame.

Common situations: Porting pandas comparison code (e.g. diffing two snapshots of a table) to Beam; copying a pandas example that uses the default align_axis='columns'; expecting compare() to produce per-column diff columns like pandas does locally.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5fc6e94aafd6033b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:4056

      if normalize:
        return result/dropped.length()
      else:
        return result

  if hasattr(pd.DataFrame, 'compare'):

    @frame_base.with_docs_from(pd.DataFrame)
    @frame_base.args_to_kwargs(pd.DataFrame)
    @frame_base.populate_defaults(pd.DataFrame)
    def compare(self, other, align_axis, keep_shape, **kwargs):
      """The default values ``align_axis=1 and ``keep_shape=False``
       are not supported, because the output columns depend on the data.
       To use ``align_axis=1``, please specify ``keep_shape=True``."""

      preserve_partition = None

      if align_axis in (1, 'columns') and not keep_shape:
        raise frame_base.WontImplementError(
          f"compare(align_axis={align_axis!r}, keep_shape={keep_shape!r}) "
          "is not allowed because the output columns depend on the data, "
          "please specify keep_shape=True.",
        reason='non-deferred-columns'
        )

      if align_axis in (1, 'columns'):
        preserve_partition = partitionings.Arbitrary()
      elif align_axis in (0, 'index'):
        preserve_partition = partitionings.Singleton()
      else:
        raise ValueError(
          "align_axis must be one of ('index', 0, 'columns', 1). "
          f"got {align_axis!r}.")


      return frame_base.DeferredFrame.wrap(
        expressions.ComputedExpression(

View on GitHub (pinned to 12126d8942)