apache/beam · warning · UserWarning

Pipeline is converting a DeferredSeries to PCollection with…

Error message

Pipeline is converting a DeferredSeries to PCollection with include_indexes=True. Note that this parameter is _not_ respected for DeferredSeries conversion. To include the index with your data, produce aDeferredDataFrame instead.

What it means

When converting a Beam DataFrame DeferredSeries to a PCollection, the include_indexes=True option cannot be honored because indexes are not carried through a Series conversion. Beam emits this UserWarning so users know the flag was silently ignored and how to get indexes back.

Solutions

1. Convert a DeferredDataFrame instead of a DeferredSeries if you need the index in the output.
2. Set include_indexes=False (or omit it) for Series conversions to silence the warning.
3. Reset the index in the dataframe pipeline (e.g. df.reset_index()) before converting so the index becomes data columns.
4. If the index is truly needed per element, carry it as an explicit Series/data column.

Example fix

// before
pcoll = convert.to_pcollection(df['col'], include_indexes=True)
// after
df_with_idx = df.reset_index()
pcoll = convert.to_pcollection(df_with_idx, include_indexes=True)

import pandas as pd
if isinstance(deferred_expr.proxy(), pd.Series) and include_indexes:
    # convert the parent DataFrame instead, or set include_indexes=False
    include_indexes = False

def is_deferred_series(obj):
    import apache_beam.dataframe.frame as f
    return isinstance(getattr(obj, 'proxy', lambda: None)(), pd.Series)

import warnings
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    pcoll = convert.to_pcollection(series)
    if any("DeferredSeries" in str(x.message) for x in w):
        # switch to DataFrame conversion path
        pass

When it happens

Trigger: Calling beam.dataframe.convert.to_pcollection(series_deferred, include_indexes=True) — or an unbatching path (maybe_unbatch/_make_unbatched_pcoll) that reaches a Series proxy with include_indexes requested.

Common situations: Migrating pandas code to Beam dataframes and keeping include_indexes=True for both DataFrames and Series; expecting .index to appear in output rows of a Series conversion.

Related errors

• include_indexes=True for a Series input. Note that this…
• Performing a numeric aggregation
• Accessing locals with @ is not yet supported…
• align_axis must be one of ('index', 0, 'columns', 1). got
• align( )

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4dd80ec0e8f0b153. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/convert.py:120

  def process_batch(self, batch: pd.Series) -> Iterable[pd.Series]:
    yield batch


def _make_unbatched_pcoll(
    pc: pvalue.PCollection, expr: expressions.Expression,
    include_indexes: bool):
  label = f"Unbatch '{expr._id}'"
  if include_indexes:
    label += " with indexes"

  if label not in UNBATCHED_CACHE:
    proxy = expr.proxy()
    shim_dofn: beam.DoFn
    if isinstance(proxy, pd.DataFrame):
      shim_dofn = DataFrameToRowsFn(proxy, include_indexes)
    elif isinstance(proxy, pd.Series):
      if include_indexes:
        warnings.warn(
            "Pipeline is converting a DeferredSeries to PCollection "
            "with include_indexes=True. Note that this parameter is "
            "_not_ respected for DeferredSeries conversion. To "
            "include the index with your data, produce a"
            "DeferredDataFrame instead.")

      shim_dofn = SeriesToElementsFn(proxy)
    else:
      raise TypeError(f"Proxy '{proxy}' has unsupported type '{type(proxy)}'")

    UNBATCHED_CACHE[label] = pc | label >> beam.ParDo(shim_dofn)

  # Note unbatched cache is keyed by the expression id as well as parameters
  # for the unbatching (i.e. include_indexes)
  return UNBATCHED_CACHE[label]


class DataFrameToRowsFn(beam.DoFn):

View on GitHub (pinned to 12126d8942)