apache/beam · error · NotImplementedError

Doubly compressed files not supported.

Error message

Doubly compressed files not supported.

What it means

apache_beam.io.gcp.gcsio raises NotImplementedError when a GCS object has both content_encoding=gzip and a content_type of application/gzip or application/x-gzip. Serving such an object causes GCS to transparently decompress it while Beam also decompresses it, leading to an infinite loop or corrupted reads (Google internal bug 203845981). Support for these doubly-compressed files is deliberately skipped until the GCS bug is fixed.

Source

Thrown at sdks/python/apache_beam/io/gcp/gcsio.py:717

    # object meets the criteria of decompressive transcoding
    # (https://cloud.google.com/storage/docs/transcoding).
    super().__init__(
        blob, chunk_size=chunk_size, retry=retry, raw_download=raw_download)
    # TODO: Remove this after
    # https://github.com/googleapis/python-storage/issues/1406 is fixed.
    # As a workaround, we manually trigger a reload here. Otherwise, an internal
    # call of reader.seek() will cause an exception if raw_download is set
    # when initializing BlobReader(),
    blob.reload()

    # TODO: Currently there is a bug in GCS server side when a client requests
    # a file with "content-encoding=gzip" and "content-type=application/gzip" or
    # "content-type=application/x-gzip", which will lead to infinite loop.
    # We skip the support of this type of files until the GCS bug is fixed.
    # Internal bug id: 203845981.
    if (blob.content_encoding == "gzip" and
        blob.content_type in ["application/gzip", "application/x-gzip"]):
      raise NotImplementedError("Doubly compressed files not supported.")

    self.enable_read_bucket_metric = enable_read_bucket_metric
    self.mode = "r"

  def read(self, size=-1):
    bytesRead = super().read(size)
    if self.enable_read_bucket_metric:
      Metrics.counter(
          self.__class__,
          "GCS_read_bytes_counter_" + self._blob.bucket.name).inc(
              len(bytesRead))
    return bytesRead


class BeamBlobWriter(BlobWriter):
  def __init__(
      self,
      blob,

View on GitHub (pinned to 12126d8942)

Solutions

1. Re-upload the file with either content_encoding or content_type adjusted so the pair (gzip + application/(x-)gzip) does not occur; e.g. keep content_encoding=gzip but set content_type to application/octet-stream.
2. Decompress the file once locally/externally and upload the plain (or singly-compressed) object, then re-run the pipeline.
3. If the payload is truly doubly compressed, decompress one layer before uploading so the stored object is single-encoded.

Example fix

// before: object uploaded with both gzip content-encoding and application/gzip content-type
gsutil -h 'Content-Type: application/gzip' -h 'Content-Encoding: gzip' cp data.gz gs://bucket/data.gz
// after
gsutil -h 'Content-Type: application/octet-stream' -h 'Content-Encoding: gzip' cp data.gz gs://bucket/data.gz

blob = bucket.get_blob(path)
if (blob.content_encoding == 'gzip' and
    blob.content_type in ('application/gzip', 'application/x-gzip')):
    raise ValueError(f'{path}: doubly compressed GCS object; fix metadata before reading')

def is_safely_readable(blob):
    return not (getattr(blob, 'content_encoding', None) == 'gzip' and
                getattr(blob, 'content_type', None) in ('application/gzip', 'application/x-gzip'))

try:
    f = gcsio.GcsIO().open(path)
except NotImplementedError as e:
    if 'Doubly compressed' in str(e):
        log.error('Re-upload %s with fixed content-type/encoding', path)
    raise

When it happens

Trigger: Opening a GCS file for reading via gcsio.GcsIO.__init__ where the object's metadata has content_encoding == 'gzip' AND content_type is 'application/gzip' or 'application/x-gzip'.

Common situations: Files uploaded by pipelines or tools that set both a gzip Content-Type header and gzip Content-Encoding (double compression), often from misconfigured upload scripts or data exported from other systems that gzip twice.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• compression
• Found no files that match {self.path!r}
• compression_type must be CompressionType object but was %s
• src and dst files do not exist. src: %s, dst: %s
• compression_type must be CompressionType object but was %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1ecf1dd48ec1d150. Report an issue: GitHub.