{"record":{"id":"299f0557d09b3b9c","repo":"apache/beam","slug":"use-p-apache-beam-dataframe-io-s","errorCode":null,"errorMessage":"Use p | apache_beam.dataframe.io.%s","messagePattern":"Use p \\| apache_beam\\.dataframe\\.io\\.(.+?)","errorType":"exception","errorClass":"WontImplementError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/pandas_top_level_functions.py","lineNumber":176,"sourceCode":"  pivot = _call_on_first_arg('pivot')\n  pivot_table = _call_on_first_arg('pivot_table')\n  set_eng_float_format = _defer_to_pandas('set_eng_float_format')\n  show_versions = _defer_to_pandas('show_versions')\n  test = frame_base.wont_implement_method(\n      pd,\n      'test',\n      explanation=\"because it is an internal pandas testing utility.\")\n  timedelta_range = _defer_to_pandas('timedelta_range')\n  to_pickle = frame_base.wont_implement_method(\n      pd, 'to_pickle', reason='order-sensitive')\n  to_datetime = _defer_to_pandas_maybe_elementwise('to_datetime')\n  notna = _call_on_first_arg('notna')\n\n  def __getattr__(self, name):\n    if name.startswith('read_'):\n\n      def func(*args, **kwargs):\n        raise frame_base.WontImplementError(\n            'Use p | apache_beam.dataframe.io.%s' % name)\n\n      return func\n    res = getattr(pd, name)\n    if _is_top_level_function(res):\n      return frame_base.not_implemented_method(name, base_type=pd)\n    else:\n      return res\n\n\npd_wrapper = DeferredPandasModule()\n","sourceCodeStart":158,"sourceCodeEnd":188,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/pandas_top_level_functions.py#L158-L188","documentation":"The Beam pandas shim module intercepts pandas top-level read_* functions. Instead of executing them eagerly, it raises WontImplementError telling you to use the deferred Beam pipeline form: p | apache_beam.dataframe.io.<name>, i.e. pipe the PCollection/deferred frame through the corresponding io transform, because eager pandas reads cannot run inside a distributed Beam pipeline.","triggerScenarios":"Calling pd.read_csv/read_json/read_parquet(...) on the apache_beam.dataframe.pandas_top_level_functions module shim (accessed via __getattr__ for names starting with 'read_'), expecting eager pandas semantics.","commonSituations":"Copy-pasting standard pandas code into a Beam pipeline; IDE autocompletion resolving to the Beam pandas shim instead of real pandas; mixing 'import pandas as pd' with Beam's deferred module in notebook pipelines.","solutions":["Use the Beam idiom: p | beam.dataframe.io.read_csv(path) instead of calling the shim function directly.","Import the real pandas module (import pandas as pd) if you truly want an eager read outside the pipeline.","Use beam.dataframe.convert.to_pcollection/to_dataframe to move data between PCollection and deferred-dataframe worlds.","Check that sys.modules/imports are not shadowing pandas with the Beam shim."],"exampleFix":"// before\ndf = pd.read_csv('input.csv')  # Beam shim raises\n// after\ndf = p | beam.dataframe.io.read_csv('input.csv')","handlingStrategy":"fallback","validationCode":"import pandas as real_pd  # ensure genuine pandas, not the Beam shim\nif hasattr(real_pd, '__beam_shim__'):\n    raise RuntimeError('pandas is shadowed by beam shim')","typeGuard":null,"tryCatchPattern":"try:\n    df = pd.read_csv(path)\nexcept frame_base.WontImplementError:\n    df = p | beam.dataframe.io.read_csv(path)","preventionTips":["Use p | beam.dataframe.io.read_* inside pipelines","Keep plain pandas imports separate from Beam deferred usage","Read Beam dataframe docs before porting pandas code"],"tags":["apache-beam","pandas","deprecated-api-usage","api-misuse"],"backgroundTag":"deprecated-api-usage","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}