apache/beam · error · TypeError

Unable to convert objects of type %s to a PCollection

Error message

Unable to convert objects of type %s to a PCollection

What it means

apache_beam.dataframe.convert.to_pcollection only accepts DeferredFrame objects (created via to_dataframe), pandas DataFrames (requiring an explicit pipeline=), or existing PCollections. If the input is none of these — e.g. a plain Python list, dict, Series, or other object — it raises this TypeError because the converter has no strategy to turn that type into a PCollection.

Source

Thrown at sdks/python/apache_beam/dataframe/convert.py:234

    # of these variables in the calling context.
    label = 'ToPCollection(%s)' % ', '.join(_var_name(e, 3) for e in dataframes)

  # Support for non-deferred dataframes.
  deferred_dataframes = []
  for ix, df in enumerate(dataframes):
    if isinstance(df, frame_base.DeferredBase):
      # TODO(robertwb): Maybe extract pipeline object?
      deferred_dataframes.append(df)
    elif isinstance(df, (pd.Series, pd.DataFrame)):
      if pipeline is None:
        raise ValueError(
            'Pipeline keyword required for non-deferred dataframe conversion.')
      deferred = pipeline | '%s_Defer%s' % (label, ix) >> beam.Create([df])
      deferred_dataframes.append(
          frame_base.DeferredFrame.wrap(
              expressions.PlaceholderExpression(df.iloc[:0], deferred)))
    else:
      raise TypeError(
          'Unable to convert objects of type %s to a PCollection' % type(df))
  dataframes = tuple(deferred_dataframes)

  def extract_input(placeholder):
    if not isinstance(placeholder._reference, pvalue.PCollection):
      raise TypeError(
          'Expression roots must have been created with to_dataframe.')
    return placeholder._reference

  placeholders = frozenset.union(
      frozenset(), *[df._expr.placeholders() for df in dataframes])

  # Exclude any dataframes that have already been converted to PCollections.
  # We only want to convert each DF expression once, then re-use.
  new_dataframes = [
      df for df in dataframes if df._expr._id not in TO_PCOLLECTION_CACHE
  ]
  if len(new_dataframes):

View on GitHub (pinned to 12126d8942)

Solutions

Check type(df); convert unsupported inputs to a pandas DataFrame first (e.g. Series.to_frame(), pd.DataFrame(list)).
For plain Python data use beam.Create directly instead of to_pcollection.
If passing a real DataFrame without pipeline=, supply the pipeline= keyword argument.
Ensure any dataframe you got from elsewhere was produced by convert.to_dataframe, not constructed ad hoc.

Example fix

// before
pcoll = convert.to_pcollection(series)
// after
pcoll = convert.to_pcollection(series.to_frame(), pipeline=pipeline)

Defensive patterns

Strategy: validation

Validate before calling

if not isinstance(obj, (pd.DataFrame, pd.Series, pvalue.PCollection)) or isinstance(obj, pd.Series) and True:
    # normalize before calling
    obj = obj.to_frame() if isinstance(obj, pd.Series) else obj

Type guard

def is_convertible_to_pcollection(obj) -> bool:
    import pandas as pd
    return isinstance(obj, (pd.DataFrame,)) or hasattr(obj, 'pipeline')

Prevention

Only pass DataFrames produced by convert.to_dataframe or concrete pd.DataFrame (with pipeline=) into to_pcollection.
Convert Series/lists/dicts to pd.DataFrame before conversion.
Keep dataframe conversions at explicit pipeline-boundary points, not scattered through logic.

When it happens

Trigger: Calling convert.to_pcollection(obj) where obj is not a pandas DataFrame, DeferredFrame, or PValue/PCollection — e.g. passing a pandas Series, a list of dicts, a numpy array, or a dict directly.

Common situations: Passing a pd.Series instead of a DataFrame after a column selection; forgetting pipeline= when passing a plain DataFrame (that raises a related error first); users assuming to_pcollection works like beam.Create on arbitrary Python values; upgrading code that previously passed lists to beam.Create and swapping in to_pcollection.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/43556d49a6381c6a. Report an issue: GitHub.