{"record":{"id":"1ecf1dd48ec1d150","repo":"apache/beam","slug":"doubly-compressed-files-not-supported","errorCode":null,"errorMessage":"Doubly compressed files not supported.","messagePattern":"Doubly compressed files not supported\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/gcsio.py","lineNumber":717,"sourceCode":"    # object meets the criteria of decompressive transcoding\n    # (https://cloud.google.com/storage/docs/transcoding).\n    super().__init__(\n        blob, chunk_size=chunk_size, retry=retry, raw_download=raw_download)\n    # TODO: Remove this after\n    # https://github.com/googleapis/python-storage/issues/1406 is fixed.\n    # As a workaround, we manually trigger a reload here. Otherwise, an internal\n    # call of reader.seek() will cause an exception if raw_download is set\n    # when initializing BlobReader(),\n    blob.reload()\n\n    # TODO: Currently there is a bug in GCS server side when a client requests\n    # a file with \"content-encoding=gzip\" and \"content-type=application/gzip\" or\n    # \"content-type=application/x-gzip\", which will lead to infinite loop.\n    # We skip the support of this type of files until the GCS bug is fixed.\n    # Internal bug id: 203845981.\n    if (blob.content_encoding == \"gzip\" and\n        blob.content_type in [\"application/gzip\", \"application/x-gzip\"]):\n      raise NotImplementedError(\"Doubly compressed files not supported.\")\n\n    self.enable_read_bucket_metric = enable_read_bucket_metric\n    self.mode = \"r\"\n\n  def read(self, size=-1):\n    bytesRead = super().read(size)\n    if self.enable_read_bucket_metric:\n      Metrics.counter(\n          self.__class__,\n          \"GCS_read_bytes_counter_\" + self._blob.bucket.name).inc(\n              len(bytesRead))\n    return bytesRead\n\n\nclass BeamBlobWriter(BlobWriter):\n  def __init__(\n      self,\n      blob,","sourceCodeStart":699,"sourceCodeEnd":735,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/gcsio.py#L699-L735","documentation":"apache_beam.io.gcp.gcsio raises NotImplementedError when a GCS object has both content_encoding=gzip and a content_type of application/gzip or application/x-gzip. Serving such an object causes GCS to transparently decompress it while Beam also decompresses it, leading to an infinite loop or corrupted reads (Google internal bug 203845981). Support for these doubly-compressed files is deliberately skipped until the GCS bug is fixed.","triggerScenarios":"Opening a GCS file for reading via gcsio.GcsIO.__init__ where the object's metadata has content_encoding == 'gzip' AND content_type is 'application/gzip' or 'application/x-gzip'.","commonSituations":"Files uploaded by pipelines or tools that set both a gzip Content-Type header and gzip Content-Encoding (double compression), often from misconfigured upload scripts or data exported from other systems that gzip twice.","solutions":["Re-upload the file with either content_encoding or content_type adjusted so the pair (gzip + application/(x-)gzip) does not occur; e.g. keep content_encoding=gzip but set content_type to application/octet-stream.","Decompress the file once locally/externally and upload the plain (or singly-compressed) object, then re-run the pipeline.","If the payload is truly doubly compressed, decompress one layer before uploading so the stored object is single-encoded."],"exampleFix":"// before: object uploaded with both gzip content-encoding and application/gzip content-type\ngsutil -h 'Content-Type: application/gzip' -h 'Content-Encoding: gzip' cp data.gz gs://bucket/data.gz\n// after\ngsutil -h 'Content-Type: application/octet-stream' -h 'Content-Encoding: gzip' cp data.gz gs://bucket/data.gz","handlingStrategy":"validation","validationCode":"blob = bucket.get_blob(path)\nif (blob.content_encoding == 'gzip' and\n    blob.content_type in ('application/gzip', 'application/x-gzip')):\n    raise ValueError(f'{path}: doubly compressed GCS object; fix metadata before reading')","typeGuard":"def is_safely_readable(blob):\n    return not (getattr(blob, 'content_encoding', None) == 'gzip' and\n                getattr(blob, 'content_type', None) in ('application/gzip', 'application/x-gzip'))","tryCatchPattern":"try:\n    f = gcsio.GcsIO().open(path)\nexcept NotImplementedError as e:\n    if 'Doubly compressed' in str(e):\n        log.error('Re-upload %s with fixed content-type/encoding', path)\n    raise","preventionTips":["Standardize upload tooling to set content_type=application/octet-stream when content_encoding=gzip","Audit existing GCS objects' metadata (gsutil ls -L) for the gzip+gzip combination before pipeline runs","Avoid double gzipping data at upload time"],"tags":["gcs","compression","unsupported-input","python","apache-beam"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}