{"record":{"id":"4dd80ec0e8f0b153","repo":"apache/beam","slug":"pipeline-is-converting-a-deferredseries-to-pcollection-with","errorCode":null,"errorMessage":"Pipeline is converting a DeferredSeries to PCollection with include_indexes=True. Note that this parameter is _not_ respected for DeferredSeries conversion. To include the index with your data, produce aDeferredDataFrame instead.","messagePattern":"Pipeline is converting a DeferredSeries to PCollection with include_indexes=True\\. Note that this parameter is _not_ respected for DeferredSeries conversion\\. To include the index with your data, produce aDeferredDataFrame instead\\.","errorType":"console","errorClass":"UserWarning","httpStatus":null,"severity":"warning","filePath":"sdks/python/apache_beam/dataframe/convert.py","lineNumber":120,"sourceCode":"  def process_batch(self, batch: pd.Series) -> Iterable[pd.Series]:\n    yield batch\n\n\ndef _make_unbatched_pcoll(\n    pc: pvalue.PCollection, expr: expressions.Expression,\n    include_indexes: bool):\n  label = f\"Unbatch '{expr._id}'\"\n  if include_indexes:\n    label += \" with indexes\"\n\n  if label not in UNBATCHED_CACHE:\n    proxy = expr.proxy()\n    shim_dofn: beam.DoFn\n    if isinstance(proxy, pd.DataFrame):\n      shim_dofn = DataFrameToRowsFn(proxy, include_indexes)\n    elif isinstance(proxy, pd.Series):\n      if include_indexes:\n        warnings.warn(\n            \"Pipeline is converting a DeferredSeries to PCollection \"\n            \"with include_indexes=True. Note that this parameter is \"\n            \"_not_ respected for DeferredSeries conversion. To \"\n            \"include the index with your data, produce a\"\n            \"DeferredDataFrame instead.\")\n\n      shim_dofn = SeriesToElementsFn(proxy)\n    else:\n      raise TypeError(f\"Proxy '{proxy}' has unsupported type '{type(proxy)}'\")\n\n    UNBATCHED_CACHE[label] = pc | label >> beam.ParDo(shim_dofn)\n\n  # Note unbatched cache is keyed by the expression id as well as parameters\n  # for the unbatching (i.e. include_indexes)\n  return UNBATCHED_CACHE[label]\n\n\nclass DataFrameToRowsFn(beam.DoFn):","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/convert.py#L102-L138","documentation":"When converting a Beam DataFrame DeferredSeries to a PCollection, the include_indexes=True option cannot be honored because indexes are not carried through a Series conversion. Beam emits this UserWarning so users know the flag was silently ignored and how to get indexes back.","triggerScenarios":"Calling beam.dataframe.convert.to_pcollection(series_deferred, include_indexes=True) — or an unbatching path (maybe_unbatch/_make_unbatched_pcoll) that reaches a Series proxy with include_indexes requested.","commonSituations":"Migrating pandas code to Beam dataframes and keeping include_indexes=True for both DataFrames and Series; expecting .index to appear in output rows of a Series conversion.","solutions":["Convert a DeferredDataFrame instead of a DeferredSeries if you need the index in the output.","Set include_indexes=False (or omit it) for Series conversions to silence the warning.","Reset the index in the dataframe pipeline (e.g. df.reset_index()) before converting so the index becomes data columns.","If the index is truly needed per element, carry it as an explicit Series/data column."],"exampleFix":"// before\npcoll = convert.to_pcollection(df['col'], include_indexes=True)\n// after\ndf_with_idx = df.reset_index()\npcoll = convert.to_pcollection(df_with_idx, include_indexes=True)","handlingStrategy":"type-guard","validationCode":"import pandas as pd\nif isinstance(deferred_expr.proxy(), pd.Series) and include_indexes:\n    # convert the parent DataFrame instead, or set include_indexes=False\n    include_indexes = False","typeGuard":"def is_deferred_series(obj):\n    import apache_beam.dataframe.frame as f\n    return isinstance(getattr(obj, 'proxy', lambda: None)(), pd.Series)","tryCatchPattern":"import warnings\nwith warnings.catch_warnings(record=True) as w:\n    warnings.simplefilter(\"always\")\n    pcoll = convert.to_pcollection(series)\n    if any(\"DeferredSeries\" in str(x.message) for x in w):\n        # switch to DataFrame conversion path\n        pass","preventionTips":["Use to_pcollection on DataFrames, not Series, when indexes matter.","Call reset_index() inside the dataframe pipeline before conversion.","Treat include_indexes as a DataFrame-only option in code review."],"tags":["python","dataframe","beam","deprecation-warning"],"backgroundTag":"unsupported-argument-combination","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}