{"record":{"id":"a6f5366b83980633","repo":"apache/beam","slug":"expression-roots-must-have-been-created-with-to-dataframe","errorCode":null,"errorMessage":"Expression roots must have been created with to_dataframe.","messagePattern":"Expression roots must have been created with to_dataframe\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/convert.py","lineNumber":240,"sourceCode":"    if isinstance(df, frame_base.DeferredBase):\n      # TODO(robertwb): Maybe extract pipeline object?\n      deferred_dataframes.append(df)\n    elif isinstance(df, (pd.Series, pd.DataFrame)):\n      if pipeline is None:\n        raise ValueError(\n            'Pipeline keyword required for non-deferred dataframe conversion.')\n      deferred = pipeline | '%s_Defer%s' % (label, ix) >> beam.Create([df])\n      deferred_dataframes.append(\n          frame_base.DeferredFrame.wrap(\n              expressions.PlaceholderExpression(df.iloc[:0], deferred)))\n    else:\n      raise TypeError(\n          'Unable to convert objects of type %s to a PCollection' % type(df))\n  dataframes = tuple(deferred_dataframes)\n\n  def extract_input(placeholder):\n    if not isinstance(placeholder._reference, pvalue.PCollection):\n      raise TypeError(\n          'Expression roots must have been created with to_dataframe.')\n    return placeholder._reference\n\n  placeholders = frozenset.union(\n      frozenset(), *[df._expr.placeholders() for df in dataframes])\n\n  # Exclude any dataframes that have already been converted to PCollections.\n  # We only want to convert each DF expression once, then re-use.\n  new_dataframes = [\n      df for df in dataframes if df._expr._id not in TO_PCOLLECTION_CACHE\n  ]\n  if len(new_dataframes):\n    new_results: dict[Any, pvalue.PCollection] = {\n        p: extract_input(p)\n        for p in placeholders\n    } | label >> transforms._DataframeExpressionsTransform(\n        {ix: df._expr\n         for (ix, df) in enumerate(new_dataframes)})","sourceCodeStart":222,"sourceCodeEnd":258,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/convert.py#L222-L258","documentation":"During to_pcollection, each placeholder expression's root reference must be a PCollection produced by convert.to_dataframe. extract_input validates this: if a placeholder's _reference is not a PCollection, the expression tree was not rooted in a to_dataframe call, so it cannot be evaluated against pipeline data, and this TypeError is thrown.","triggerScenarios":"Building a DeferredFrame/expression whose placeholder root was created by something other than convert.to_dataframe — e.g. manually wrapping expressions.PlaceholderExpression around a dataframe not backed by a PCollection, or calling to_pcollection on a deferred frame obtained outside the to_dataframe path.","commonSituations":"Hand-constructing apache_beam.dataframe expressions for advanced use; mixing a deferred frame from one pipeline with another pipeline's to_pcollection call; writing library code that wraps user expressions without verifying their provenance.","solutions":["Always create the deferred frame with convert.to_dataframe(pcollection) before converting back with to_pcollection.","Verify placeholder._reference is a pvalue.PCollection before calling to_pcollection.","Rebuild the expression tree starting from a proper to_dataframe call rather than manual PlaceholderExpression construction."],"exampleFix":"// before\ndf = frame_base.DeferredFrame.wrap(expressions.PlaceholderExpression(pd.DataFrame()))\npcoll = convert.to_pcollection(df)\n// after\ndf = convert.to_dataframe(pcoll_in)\npcoll_out = convert.to_pcollection(df)","handlingStrategy":"validation","validationCode":"from apache_beam.pvalue import PCollection\nassert isinstance(expr.root_reference(), PCollection), 'root must come from to_dataframe()'","typeGuard":"def rooted_in_to_dataframe(placeholder) -> bool:\n    from apache_beam.pvalue import PCollection\n    return isinstance(placeholder._reference, PCollection)","tryCatchPattern":"try:\n    pcoll = convert.to_pcollection(df)\nexcept TypeError as e:\n    if 'to_dataframe' in str(e):\n        df = convert.to_dataframe(source_pcoll)\n        pcoll = convert.to_pcollection(df)\n    else:\n        raise","preventionTips":["Always pair convert.to_dataframe with convert.to_pcollection from the same pipeline.","Never hand-build expressions.PlaceholderExpression unless you know the reference is a PCollection.","Round-trip only frames whose provenance is a to_dataframe call."],"tags":["python","apache-beam","dataframe-api","expression-tree"],"backgroundTag":"incompatible-source-type","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}