{"record":{"id":"ca9f8333766b7983","repo":"apache/beam","slug":"compression","errorCode":null,"errorMessage":"compression","messagePattern":"compression","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/io.py","lineNumber":263,"sourceCode":"  if isinstance(df.index, pd.RangeIndex):\n    return df.set_index(df.index + offset)\n  else:\n    return df\n\n\nclass _ReadFromPandas(beam.PTransform):\n  def __init__(\n      self,\n      reader,\n      path,\n      args,\n      kwargs,\n      binary=True,\n      incremental=False,\n      splitter=False,\n      filename_column=None):\n    if 'compression' in kwargs:\n      raise NotImplementedError('compression')\n    if not isinstance(path, str):\n      raise frame_base.WontImplementError('non-deferred')\n    self.reader = reader\n    self.path = path\n    self.args = args\n    self.kwargs = kwargs\n    self.binary = binary\n    self.incremental = incremental\n    self.splitter = splitter\n    self.filename_column = filename_column\n\n  def expand(self, root):\n    paths_pcoll = root | beam.Create([self.path])\n    match = io.filesystems.FileSystems.match([self.path], limits=[1])[0]\n    if not match.metadata_list:\n      # TODO(https://github.com/apache/beam/issues/20858): This should be\n      # allowed for streaming pipelines if user provides an explicit schema.\n      raise FileNotFoundError(f\"Found no files that match {self.path!r}\")","sourceCodeStart":245,"sourceCodeEnd":281,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/io.py#L245-L281","documentation":"The _ReadFromPandas constructor rejects any 'compression' keyword with NotImplementedError, because Beam reads files in chunks over the Beam FileSystems layer and does not implement pandas-side compression handling for these reads.","triggerScenarios":"Calling read_csv/read_json/read_fwf with compression='gzip' (or 'zip', 'bz2') in kwargs, or reading files via a path configured with a compression option.","commonSituations":"Reading gzipped CSVs the pandas way; configs copied from pandas pipelines that set compression automatically by extension.","solutions":["Remove the compression kwarg and pre-decompress the files before the pipeline (or store uncompressed).","For gzipped text, note that Beam FileSystems handles gzip transparently for splittable/plain reads only if the underlying format allows; use apache_beam.io text sources with compression if needed.","Drop to pandas locally or use beam.io.ReadFromText(compression_type=...) for compressed inputs."],"exampleFix":"// before\nread_csv('data.csv.gz', compression='gzip')\n// after\ngunzip data.csv.gz  # then\nread_csv('data.csv')","handlingStrategy":"validation","validationCode":"if 'compression' in kwargs:\n    raise ValueError('Beam dataframe IO does not support compression kwarg; pre-decompress inputs')","typeGuard":null,"tryCatchPattern":"try:\n    df = read_csv(path, **kwargs)\nexcept NotImplementedError as e:\n    if 'compression' in str(e):\n        kwargs.pop('compression', None)\n        path = decompress_to_temp(path)\n        df = read_csv(path, **kwargs)\n    else:\n        raise","preventionTips":["Pre-decompress or store plain-text inputs for Beam DataFrame reads","Never forward pandas compression kwargs","Document storage format expectations in pipeline config"],"tags":["python","apache-beam","dataframe","compression"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}