apache/beam · error · WontImplementError

other must be a DeferredDataFrame or DeferredSeries…

Error message

other must be a DeferredDataFrame or DeferredSeries instance. Passing a concrete list or numpy array is not supported. Those types have no index and must be joined based on the order of the data.

What it means

DeferredDataFrame.dot() requires the operand to be another DeferredDataFrame or DeferredSeries because matrix multiplication joins on indexes. Passing a concrete list or numpy array would require positional (order-based) joining, which is not deterministic in a distributed pipeline, so WontImplementError (reason 'order-sensitive') is raised.

Solutions

  1. Wrap the operand in a DeferredSeries/DeferredDataFrame, e.g. beam.dataframe.Series from a PCollection with the right index
  2. Convert the list to a pandas Series and then to a deferred frame via the Beam dataframe API
  3. Compute the dot product with non-deferred pandas/numpy on collected data if the data is small enough

Example fix

// before
df.dot([1, 2, 3])
// after
weights = beam.dataframe.Series(index_pd_series_of_weights)  # deferred, with matching index
df.dot(weights)
Defensive patterns

Strategy: type-guard

Validate before calling

assert isinstance(other, (DeferredDataFrame, DeferredSeries)), f'df.dot: unsupported operand {type(other)}'

Type guard

from apache_beam.dataframe.frames import DeferredDataFrame, DeferredSeries
def is_deferred_operand(other):
    return isinstance(other, (DeferredDataFrame, DeferredSeries))

Try / catch

from apache_beam.dataframe import frame_base
try:
    return df.dot(other)
except frame_base.WontImplementError:
    raise TypeError('Convert operand to DeferredDataFrame/DeferredSeries before dot()')

Prevention

When it happens

Trigger: Calling df.dot([1, 2, 3]) or df.dot(np.array([...])), i.e. any other argument that is not a DeferredDataFrame/DeferredSeries (and isn't wrapped as one).

Common situations: Porting pandas snippets that compute dot products against raw lists/arrays; converting numeric code (linear algebra, weighted sums) to Beam dataframes.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ae82537b9230f01e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/frames.py:1559

  @frame_base.with_docs_from(pd.DataFrame)
  def dot(self, other):
    """``other`` must be a :class:`DeferredDataFrame` or :class:`DeferredSeries`
    instance. Computing the dot product with an array-like is not supported
    because it is order-sensitive."""
    left = self._expr
    if isinstance(other, DeferredSeries):
      right = expressions.ComputedExpression(
          'to_dataframe',
          pd.DataFrame, [other._expr],
          requires_partition_by=partitionings.Arbitrary(),
          preserves_partition_by=partitionings.Arbitrary())
      right_is_series = True
    elif isinstance(other, DeferredDataFrame):
      right = other._expr
      right_is_series = False
    else:
      raise frame_base.WontImplementError(
          "other must be a DeferredDataFrame or DeferredSeries instance. "
          "Passing a concrete list or numpy array is not supported. Those "
          "types have no index and must be joined based on the order of the "
          "data.",
          reason="order-sensitive")

    dots = expressions.ComputedExpression(
        'dot',
        # Transpose so we can sum across rows.
        (lambda left, right: pd.DataFrame(left @ right).T),
        [left, right],
        requires_partition_by=partitionings.Index())
    with expressions.allow_non_parallel_operations(True):
      sums = expressions.ComputedExpression(
          'sum',
          lambda dots: dots.sum(),  #
          [dots],
          requires_partition_by=partitionings.Singleton())

View on GitHub (pinned to 12126d8942)