{"record":{"id":"5fc6e94aafd6033b","repo":"apache/beam","slug":"compare-align-axis-align-axis-r-keep-shape-keep-shape-r-is","errorCode":null,"errorMessage":"compare(align_axis={align_axis!r}, keep_shape={keep_shape!r}) is not allowed because the output columns depend on the data, please specify keep_shape=True.","messagePattern":"compare\\(align_axis=(.+?), keep_shape=(.+?)\\) is not allowed because the output columns depend on the data, please specify keep_shape=True\\.","errorType":"exception","errorClass":"WontImplementError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/frames.py","lineNumber":4056,"sourceCode":"      if normalize:\n        return result/dropped.length()\n      else:\n        return result\n\n  if hasattr(pd.DataFrame, 'compare'):\n\n    @frame_base.with_docs_from(pd.DataFrame)\n    @frame_base.args_to_kwargs(pd.DataFrame)\n    @frame_base.populate_defaults(pd.DataFrame)\n    def compare(self, other, align_axis, keep_shape, **kwargs):\n      \"\"\"The default values ``align_axis=1 and ``keep_shape=False``\n       are not supported, because the output columns depend on the data.\n       To use ``align_axis=1``, please specify ``keep_shape=True``.\"\"\"\n\n      preserve_partition = None\n\n      if align_axis in (1, 'columns') and not keep_shape:\n        raise frame_base.WontImplementError(\n          f\"compare(align_axis={align_axis!r}, keep_shape={keep_shape!r}) \"\n          \"is not allowed because the output columns depend on the data, \"\n          \"please specify keep_shape=True.\",\n        reason='non-deferred-columns'\n        )\n\n      if align_axis in (1, 'columns'):\n        preserve_partition = partitionings.Arbitrary()\n      elif align_axis in (0, 'index'):\n        preserve_partition = partitionings.Singleton()\n      else:\n        raise ValueError(\n          \"align_axis must be one of ('index', 0, 'columns', 1). \"\n          f\"got {align_axis!r}.\")\n\n\n      return frame_base.DeferredFrame.wrap(\n        expressions.ComputedExpression(","sourceCodeStart":4038,"sourceCodeEnd":4074,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/frames.py#L4038-L4074","documentation":"apache_beam.dataframe raises WontImplementError for DataFrame.compare() when align_axis is 1/'columns' and keep_shape is not True. In Beam's deferred model, the output columns would be determined by comparing the actual data at execution time, which the pipeline cannot do ahead of time. The library forbids data-dependent columns and asks you to pass keep_shape=True so the column set is fixed.","triggerScenarios":"Calling df.compare(other) or df.compare(other, align_axis=1) / align_axis='columns' without keep_shape=True on a deferred Beam DataFrame.","commonSituations":"Porting pandas comparison code (e.g. diffing two snapshots of a table) to Beam; copying a pandas example that uses the default align_axis='columns'; expecting compare() to produce per-column diff columns like pandas does locally.","solutions":["Pass keep_shape=True: df.compare(other, align_axis='columns', keep_shape=True).","Use the default align_axis=0 ('rows') so the comparison produces rows instead of data-dependent columns.","Restructure the comparison manually (e.g. select and rename columns yourself) so the output schema is static.","If a data-dependent column layout is truly required, run that step in plain pandas outside the Beam pipeline."],"exampleFix":"// before\ndiff = df.compare(other_df)\n\n// after\ndiff = df.compare(other_df, align_axis='columns', keep_shape=True)","handlingStrategy":"validation","validationCode":"if align_axis in (1, 'columns') and not keep_shape:\n    raise ValueError(\"Use keep_shape=True with align_axis=1 in Beam dataframes\")","typeGuard":null,"tryCatchPattern":"try:\n    diff = df.compare(other, align_axis='columns')\nexcept apachebeam.WontImplementError:\n    diff = df.compare(other, align_axis='columns', keep_shape=True)","preventionTips":["Never call compare() with align_axis=1/'columns' on deferred frames without keep_shape=True.","Prefer align_axis=0 (row-wise diffs) in Beam pipelines.","Review ported pandas code for column-creating operations before moving them into Beam."],"tags":["apache-beam","dataframe","deferred-columns","pandas"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}