apache/beam · error · NotImplementedError

cross join is not yet implemented…

Error message

cross join is not yet implemented (https://github.com/apache/beam/issues/20318)

What it means

DeferredFrame.merge() does not implement how='cross' (Cartesian product of two frames) because a full cross join is expensive and had no partitioning strategy at the time; Beam raises NotImplementedError referencing GitHub issue 20318.

Solutions

Add a constant join key to both frames and merge with how='inner' on that key (beware data volume)
Materialize frames with to_pandas() and perform the cross join in pandas
Build the cross product with a custom Beam flatten/DoFn transform

Example fix

// before
merged = a.beam.merge(b.beam, how='cross')
// after
a = a.assign(_k=1).beam
b = b.assign(_k=1).beam
merged = a.merge(b, how='inner', on='_k').drop(columns='_k')

Defensive patterns

Strategy: fallback

Validate before calling

if kwargs.get('how') == 'cross':
    raise ValueError('cross merge unsupported on Beam deferred frames')

Try / catch

try:
    merged = a.merge(b, on=key)
except NotImplementedError:
    merged = a.to_pandas().merge(b.to_pandas(), how='cross')

Prevention

Avoid how='cross' in pipeline code; use a sentinel key merge
Estimate cross-join size first to avoid explode-scale mistakes
Check the Beam issue tracker for join capability before designing pipelines

When it happens

Trigger: Calling left.beam.merge(right.beam, how='cross') on deferred Beam frames

Common situations: Porting pandas cross-join code (e.g. generating all pairwise combinations) to Beam

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/eef8a8a5cd3033d8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:3499

    move the join key for one of your columns to the index to avoid this issue.
    For an example see the enrich pipeline in
    :mod:`apache_beam.examples.dataframe.taxiride`.

    ``how="cross"`` is not yet supported.
    """
    self_proxy = self._expr.proxy()
    right_proxy = right._expr.proxy()
    # Validate with a pandas call.
    _ = self_proxy.merge(
        right_proxy,
        on=on,
        left_on=left_on,
        right_on=right_on,
        left_index=left_index,
        right_index=right_index,
        **kwargs)
    if kwargs.get('how', None) == 'cross':
      raise NotImplementedError(
        "cross join is not yet implemented "
        "(https://github.com/apache/beam/issues/20318)")
    if not any([on, left_on, right_on, left_index, right_index]):
      on = [col for col in self_proxy.columns if col in right_proxy.columns]
    if not left_on:
      left_on = on
    if left_on and not isinstance(left_on, list):
      left_on = [left_on]
    if not right_on:
      right_on = on
    if right_on and not isinstance(right_on, list):
      right_on = [right_on]

    if left_index:
      indexed_left = self
    else:
      indexed_left = self.set_index(left_on, drop=False)

View on GitHub (pinned to 12126d8942)