apache/beam · error · TypeError

Expression roots must have been created with to_dataframe.

Error message

Expression roots must have been created with to_dataframe.

What it means

During to_pcollection, each placeholder expression's root reference must be a PCollection produced by convert.to_dataframe. extract_input validates this: if a placeholder's _reference is not a PCollection, the expression tree was not rooted in a to_dataframe call, so it cannot be evaluated against pipeline data, and this TypeError is thrown.

Source

Thrown at sdks/python/apache_beam/dataframe/convert.py:240

    if isinstance(df, frame_base.DeferredBase):
      # TODO(robertwb): Maybe extract pipeline object?
      deferred_dataframes.append(df)
    elif isinstance(df, (pd.Series, pd.DataFrame)):
      if pipeline is None:
        raise ValueError(
            'Pipeline keyword required for non-deferred dataframe conversion.')
      deferred = pipeline | '%s_Defer%s' % (label, ix) >> beam.Create([df])
      deferred_dataframes.append(
          frame_base.DeferredFrame.wrap(
              expressions.PlaceholderExpression(df.iloc[:0], deferred)))
    else:
      raise TypeError(
          'Unable to convert objects of type %s to a PCollection' % type(df))
  dataframes = tuple(deferred_dataframes)

  def extract_input(placeholder):
    if not isinstance(placeholder._reference, pvalue.PCollection):
      raise TypeError(
          'Expression roots must have been created with to_dataframe.')
    return placeholder._reference

  placeholders = frozenset.union(
      frozenset(), *[df._expr.placeholders() for df in dataframes])

  # Exclude any dataframes that have already been converted to PCollections.
  # We only want to convert each DF expression once, then re-use.
  new_dataframes = [
      df for df in dataframes if df._expr._id not in TO_PCOLLECTION_CACHE
  ]
  if len(new_dataframes):
    new_results: dict[Any, pvalue.PCollection] = {
        p: extract_input(p)
        for p in placeholders
    } | label >> transforms._DataframeExpressionsTransform(
        {ix: df._expr
         for (ix, df) in enumerate(new_dataframes)})

View on GitHub (pinned to 12126d8942)

Solutions

Always create the deferred frame with convert.to_dataframe(pcollection) before converting back with to_pcollection.
Verify placeholder._reference is a pvalue.PCollection before calling to_pcollection.
Rebuild the expression tree starting from a proper to_dataframe call rather than manual PlaceholderExpression construction.

Example fix

// before
df = frame_base.DeferredFrame.wrap(expressions.PlaceholderExpression(pd.DataFrame()))
pcoll = convert.to_pcollection(df)
// after
df = convert.to_dataframe(pcoll_in)
pcoll_out = convert.to_pcollection(df)

Defensive patterns

Strategy: validation

Validate before calling

from apache_beam.pvalue import PCollection
assert isinstance(expr.root_reference(), PCollection), 'root must come from to_dataframe()'

Type guard

def rooted_in_to_dataframe(placeholder) -> bool:
    from apache_beam.pvalue import PCollection
    return isinstance(placeholder._reference, PCollection)

Try / catch

try:
    pcoll = convert.to_pcollection(df)
except TypeError as e:
    if 'to_dataframe' in str(e):
        df = convert.to_dataframe(source_pcoll)
        pcoll = convert.to_pcollection(df)
    else:
        raise

Prevention

Always pair convert.to_dataframe with convert.to_pcollection from the same pipeline.
Never hand-build expressions.PlaceholderExpression unless you know the reference is a PCollection.
Round-trip only frames whose provenance is a to_dataframe call.

When it happens

Trigger: Building a DeferredFrame/expression whose placeholder root was created by something other than convert.to_dataframe — e.g. manually wrapping expressions.PlaceholderExpression around a dataframe not backed by a PCollection, or calling to_pcollection on a deferred frame obtained outside the to_dataframe path.

Common situations: Hand-constructing apache_beam.dataframe expressions for advanced use; mixing a deferred frame from one pipeline with another pipeline's to_pcollection call; writing library code that wraps user expressions without verifying their provenance.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a6f5366b83980633. Report an issue: GitHub.