apache/beam · error · WontImplementError

non-deferred

Error message

non-deferred

What it means

_ReadFromPandas (used by read_csv/read_json/etc. in apache_beam.dataframe.io) requires path to be a str so the source can be deferred as a file reference. If path is not a string (e.g. a file-like object, bytes, or pathlib.Path), the constructor raises WontImplementError('non-deferred') because Beam cannot represent a non-string source in its deferred dataframe pipeline.

Solutions

1. Pass a plain str path (or str-compatible path) to the read_* function.
2. Convert pathlib.Path with str(path) before calling.
3. Materialize any file-like object to disk first, then pass its filename string.
4. Read the data with plain pandas, then wrap it: beam.dataframe.convert.to_pcollection(pd_df).

Example fix

// before
with open('data.csv') as f:
  df = io.read_csv(f)
// after
df = io.read_csv('data.csv')

if not isinstance(path, str):
    raise TypeError(f'expected str path, got {type(path).__name__}')

def is_str_path(path):
    return isinstance(path, str)

try:
    df = beam_df.io.read_csv(path)
except frame_base.WontImplementError:
    df = beam_df.io.read_csv(str(path))

When it happens

Trigger: Calling beam.dataframe.io.read_csv(path) / read_json / read_parquet with a file-like object, BytesIO, bytes, pathlib.Path, or URL object instead of a str path; also if 'compression' is passed a NotImplementedError is raised in the same __init__.

Common situations: Developers used to pandas' acceptance of file handles pass an open file object; using pathlib.Path results from os.path operations; downloading logic hands in a stream instead of saving to a file path first.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• len(df) is not currently supported because it produces a…
• A schema is required to write non-schema'd data.
• A sink must inherit iobase.Sink, iobase.NativeSink, or be a…
• Accessing a DeferredSeries with an iterator is sensitive to…
• Accessing an item by an integer key is order sensitive for…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1a3678e24efe5fc0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/io.py:265

  else:
    return df


class _ReadFromPandas(beam.PTransform):
  def __init__(
      self,
      reader,
      path,
      args,
      kwargs,
      binary=True,
      incremental=False,
      splitter=False,
      filename_column=None):
    if 'compression' in kwargs:
      raise NotImplementedError('compression')
    if not isinstance(path, str):
      raise frame_base.WontImplementError('non-deferred')
    self.reader = reader
    self.path = path
    self.args = args
    self.kwargs = kwargs
    self.binary = binary
    self.incremental = incremental
    self.splitter = splitter
    self.filename_column = filename_column

  def expand(self, root):
    paths_pcoll = root | beam.Create([self.path])
    match = io.filesystems.FileSystems.match([self.path], limits=[1])[0]
    if not match.metadata_list:
      # TODO(https://github.com/apache/beam/issues/20858): This should be
      # allowed for streaming pipelines if user provides an explicit schema.
      raise FileNotFoundError(f"Found no files that match {self.path!r}")
    first_path = match.metadata_list[0].path
    with io.filesystems.FileSystems.open(first_path) as handle:

View on GitHub (pinned to 12126d8942)