apache/beam · error · ValueError

Cannot infer a proxy because the input PCollection does not

Error message

Cannot infer a proxy because the input PCollection does not have a schema defined. Please make sure a schema type is specified for the input PCollection, or provide a proxy.

What it means

beam.dataframe.convert.to_dataframe converts a PCollection to a deferred pandas DataFrame. To build the placeholder DataFrame it needs to know the element schema; if the PCollection has element_type None (no schema inferred) and no explicit proxy is supplied, a ValueError is raised telling you to define a schema or pass a proxy.

Source

Thrown at sdks/python/apache_beam/dataframe/convert.py:57

    pcoll: pvalue.PCollection,
    proxy: Optional[pd.core.generic.NDFrame] = None,
    label: Optional[str] = None,
) -> frame_base.DeferredFrame:
  """Converts a PCollection to a deferred dataframe-like object, which can
  manipulated with pandas methods like `filter` and `groupby`.

  For example, one might write::

    pcoll = ...
    df = to_dataframe(pcoll, proxy=...)
    result = df.groupby('col').sum()
    pcoll_result = to_pcollection(result)

  A proxy object must be given if the schema for the PCollection is not known.
  """
  if proxy is None:
    if pcoll.element_type is None:
      raise ValueError(
          "Cannot infer a proxy because the input PCollection does not have a "
          "schema defined. Please make sure a schema type is specified for "
          "the input PCollection, or provide a proxy.")
    # If no proxy is given, assume this is an element-wise schema-aware
    # PCollection that needs to be batched.
    if label is None:
      # Attempt to come up with a reasonable, stable label by retrieving
      # the name of these variables in the calling context.
      label = 'BatchElements(%s)' % _var_name(pcoll, 2)
    proxy = generate_proxy(pcoll.element_type)

    shim_dofn: beam.DoFn
    if isinstance(proxy, pd.DataFrame):
      shim_dofn = RowsToDataFrameFn()
    elif isinstance(proxy, pd.Series):
      shim_dofn = ElementsToSeriesFn()
    else:
      raise AssertionError("Unknown proxy type: %s" % proxy)

View on GitHub (pinned to 12126d8942)

Solutions

1. Provide an explicit proxy: convert.to_dataframe(pcoll, proxy=pd.DataFrame({'col': pd.Series([], dtype=...)})).
2. Give the PCollection an element type/schema (e.g. with_row_type, typed beam.Row, or type hints on the producing DoFn).
3. Ensure element types aren't erased by lambdas — use named functions or explicit type hints.
4. If elements are plain dicts, annotate the PCollection with a schema via pc | beam.Map(...).with_output_type(SomeRow).

Example fix

// before
df = convert.to_dataframe(untyped_pcoll)
// after
proxy = pd.DataFrame({'col_a': pd.Series(dtype='int64'), 'col_b': pd.Series(dtype=float)})
df = convert.to_dataframe(untyped_pcoll, proxy=proxy)

if pcoll.element_type is None and proxy is None:
    raise ValueError('to_dataframe requires a schema-typed PCollection or an explicit proxy')

try:
    df = convert.to_dataframe(pcoll)
except ValueError as e:
    if 'Cannot infer a proxy' in str(e):
        df = convert.to_dataframe(pcoll, proxy=build_proxy_from_sample(pcoll))
    else:
        raise

When it happens

Trigger: Calling convert.to_dataframe(pcoll) where pcoll was created from untyped elements (e.g. beam.Create of dicts without element type, JSON-parsed rows) and proxy=None.

Common situations: Reading untyped sources (text/JSON/Avro without schema) then using the DataFrame API; pipelines losing type hints through lambdas; migrating pipelines from non-schema PCollections to dataframe transforms.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• include_indexes=True for a Series input. Note that this para
• Proxy '{proxy}' has unsupported type '{type(proxy)}'
• concat(ignore_index)
• concat(levels)
• Encountered an unnamed index. Cannot convert to a schema-awa

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/db8db54081e06249. Report an issue: GitHub.