apache/hadoop · error · IOException

Cannot read GZIP encoded files - content encoding support is

Error message

Cannot read GZIP encoded files - content encoding support is disabled.

What it means

GoogleCloudStorageClientReadChannel.initMetadata throws when the object's metadata Content-Encoding contains "gzip" but the read configuration has gzip encoding support disabled. Support is opt-in because a gzip-encoded object's decompressed size is unknown: when enabled the channel treats objectSize as Long.MAX_VALUE and disables range reads; when disabled (the default) reads are refused outright.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageClientReadChannel.java:90

  GoogleCloudStorageClientReadChannel(
      Storage storage,
      GoogleCloudStorageItemInfo itemInfo,
      GoogleHadoopFileSystemConfiguration config)
      throws IOException {
    validate(itemInfo);
    this.storage = storage;
    this.resourceId =
        new StorageResourceId(
            itemInfo.getBucketName(), itemInfo.getObjectName(), itemInfo.getContentGeneration());
    this.contentReadChannel = new ContentReadChannel(config, resourceId);
    initMetadata(itemInfo.getContentEncoding(), itemInfo.getSize());
    this.config = config;
  }

  protected void initMetadata(@Nullable String encoding, long sizeFromMetadata) throws IOException {
    gzipEncoded = nullToEmpty(encoding).contains(GZIP_ENCODING);
    if (gzipEncoded && !config.isGzipEncodingSupportEnabled()) {
      throw new IOException(
          "Cannot read GZIP encoded files - content encoding support is disabled.");
    }
    objectSize = gzipEncoded ? Long.MAX_VALUE : sizeFromMetadata;
  }

  @Override
  public int read(ByteBuffer dst) throws IOException {
    throwIfNotOpen();

    // Don't try to read if the buffer has no space.
    if (dst.remaining() == 0) {
      return 0;
    }
    LOG.trace(
        "Reading {} bytes at {} position from '{}'", dst.remaining(), currentPosition, resourceId);
    if (currentPosition == objectSize) {
      return -1;
    }

View on GitHub (pinned to 2add963021)

Solutions

Set fs.gs.inputstream.support.gzip.encoding.enable=true in core-site.xml (or enableGzipEncodingSupport(true) on GoogleCloudStorageOptions) on every reader node.
Re-upload the data as plain bytes in a .gz file (no Content-Encoding header) and read it through Hadoop compression codecs, which also restores random access.
Remove the gzip Content-Encoding from the object metadata if the payload is not actually gzip.

Example fix

# before (core-site.xml)
<property><name>fs.gs.inputstream.support.gzip.encoding.enable</name><value>false</value></property>

# after
<property><name>fs.gs.inputstream.support.gzip.encoding.enable</name><value>true</value></property>

Defensive patterns

Strategy: validation

Validate before calling

GoogleCloudStorageItemInfo info = gcs.getItemInfo(resourceId);
boolean gzipEncoded = info.getContentEncoding() != null
    && info.getContentEncoding().contains("gzip");
if (gzipEncoded && !options.isGzipEncodingSupportEnabled()) {
  throw new IllegalStateException(
      "Enable fs.gs.inputstream.support.gzip.encoding.enable or re-upload without Content-Encoding: gzip");
}
try (SeekableByteChannel ch = gcs.open(info)) { /* ... */ }

Try / catch

try (SeekableByteChannel ch = gcs.open(info)) { /* read */ } catch (IOException e) { if (e.getMessage().contains("GZIP")) { /* enable config or switch to codec-based .gz read */ } else throw e; }

Prevention

Set fs.gs.inputstream.support.gzip.encoding.enable=true on reader nodes when ingesting gzip-encoded objects.
Prefer plain .gz objects + Hadoop compression codecs when random access is needed.
After connector upgrades, re-check config key names (fs.gs.inputstream.support.gzip.encoding.enable).

When it happens

Trigger: Opening a read channel on an object uploaded with Content-Encoding: gzip (e.g. `gcloud storage cp -j`, gsutil -j, or setContentEncoding("gzip") in the storage client) while fs.gs.inputstream.support.gzip.encoding.enable is false (the default) or GoogleCloudStorageOptions was built without enableGzipEncodingSupport(true).

Common situations: Data pipelines that gzip objects transparently on upload; migrating from gsutil-based ingest to the Hadoop connector; older connector versions used a different key name (fs.gs.gzip.encoding.support.enabled) so configs silently lose the setting after upgrade.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b037cd9908866438. Report an issue: GitHub.