apache/beam · error · NotImplementedError

compression

Error message

compression

What it means

The _ReadFromPandas constructor rejects any 'compression' keyword with NotImplementedError, because Beam reads files in chunks over the Beam FileSystems layer and does not implement pandas-side compression handling for these reads.

Source

Thrown at sdks/python/apache_beam/dataframe/io.py:263

  if isinstance(df.index, pd.RangeIndex):
    return df.set_index(df.index + offset)
  else:
    return df


class _ReadFromPandas(beam.PTransform):
  def __init__(
      self,
      reader,
      path,
      args,
      kwargs,
      binary=True,
      incremental=False,
      splitter=False,
      filename_column=None):
    if 'compression' in kwargs:
      raise NotImplementedError('compression')
    if not isinstance(path, str):
      raise frame_base.WontImplementError('non-deferred')
    self.reader = reader
    self.path = path
    self.args = args
    self.kwargs = kwargs
    self.binary = binary
    self.incremental = incremental
    self.splitter = splitter
    self.filename_column = filename_column

  def expand(self, root):
    paths_pcoll = root | beam.Create([self.path])
    match = io.filesystems.FileSystems.match([self.path], limits=[1])[0]
    if not match.metadata_list:
      # TODO(https://github.com/apache/beam/issues/20858): This should be
      # allowed for streaming pipelines if user provides an explicit schema.
      raise FileNotFoundError(f"Found no files that match {self.path!r}")

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove the compression kwarg and pre-decompress the files before the pipeline (or store uncompressed).
2. For gzipped text, note that Beam FileSystems handles gzip transparently for splittable/plain reads only if the underlying format allows; use apache_beam.io text sources with compression if needed.
3. Drop to pandas locally or use beam.io.ReadFromText(compression_type=...) for compressed inputs.

Example fix

// before
read_csv('data.csv.gz', compression='gzip')
// after
gunzip data.csv.gz  # then
read_csv('data.csv')

if 'compression' in kwargs:
    raise ValueError('Beam dataframe IO does not support compression kwarg; pre-decompress inputs')

try:
    df = read_csv(path, **kwargs)
except NotImplementedError as e:
    if 'compression' in str(e):
        kwargs.pop('compression', None)
        path = decompress_to_temp(path)
        df = read_csv(path, **kwargs)
    else:
        raise

When it happens

Trigger: Calling read_csv/read_json/read_fwf with compression='gzip' (or 'zip', 'bz2') in kwargs, or reading files via a path configured with a compression option.

Common situations: Reading gzipped CSVs the pandas way; configs copied from pandas pipelines that set compression automatically by extension.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Cannot specify both 'labels' and 'index'/'columns'
• axis must be one of (0, 1, 'index', 'columns'), got '%s'
• groupby(as_index=False)
• You have to supply one of 'by' and 'level'
• label

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ca9f8333766b7983. Report an issue: GitHub.