apache/hadoop · error · IOException

listing object '%s' failed.

Error message

listing object '%s' failed.

What it means

GoogleCloudStorage.listDirectory wraps every StorageException from the underlying GcsListOperation (current-directory listing) in an IOException: "listing object '<bucket, prefix>' failed.". The real reason is in the nested cause: 404 bucket missing, 401/403 credentials or scopes, 429/5xx quota and transient faults, or network/proxy failures. The prefix must end with '/' or an earlier checkArgument fails with a different exception.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java:911

  List<GoogleCloudStorageItemInfo> listDirectory(String bucketName, String objectNamePrefix)
      throws IOException {
    checkArgument(
        objectNamePrefix == null || objectNamePrefix.endsWith("/"),
        String.format("%s should end with /", objectNamePrefix));

    try {
      List<Blob> blobs = new GcsListOperation.Builder(bucketName, objectNamePrefix, storage)
          .forCurrentDirectoryListing().build()
          .execute();

      ListOperationResult result = new ListOperationResult();
      for (Blob blob : blobs) {
        result.add(blob);
      }

      return result.getItems();
    } catch (StorageException e) {
      throw new IOException(
          String.format("listing object '%s' failed.", BlobId.of(bucketName, objectNamePrefix)),
          e);
    }
  }

  void compose(
      String bucketName, List<String> sources, String destination, String contentType)
      throws IOException {
    LOG.trace("compose({}, {}, {}, {})", bucketName, sources, destination, contentType);
    List<StorageResourceId> sourceIds =
        sources.stream()
            .map(objectName -> new StorageResourceId(bucketName, objectName))
            .collect(Collectors.toList());
    StorageResourceId destinationId = new StorageResourceId(bucketName, destination);
    CreateObjectOptions options =
        CreateObjectOptions.DEFAULT_OVERWRITE.toBuilder()
            .setContentType(contentType)
            .setEnsureEmptyObjectsMetadataMatch(false)

View on GitHub (pinned to 2add963021)

Solutions

  1. Unwrap and inspect the cause StorageException code: 404 -> bucket gone/typo'd, 401/403 -> fix credentials/scopes, 429/5xx -> retry with backoff.
  2. Verify the bucket exists and the caller has storage.objects.list permission (e.g. `gcloud storage ls gs://bucket` with the same identity).
  3. Retry listing on 429/5xx with exponential backoff and jitter; these are transient by contract.
  4. Check proxy, DNS, and VPC-SC configuration if every storage call fails, not just listings.

Example fix

// before
List<GoogleCloudStorageItemInfo> items = gcs.listDirectory(bucket, prefix); // IOException, cause hidden

// after
try {
  List<GoogleCloudStorageItemInfo> items = gcs.listDirectory(bucket, prefix);
} catch (IOException e) {
  if (e.getCause() instanceof StorageException se) {
    int code = se.getCode(); // 404, 403, 429, 5xx ...
  }
  throw e;
}
Defensive patterns

Strategy: retry

Validate before calling

// Cheap pre-flight: confirm the bucket is listable with the same credentials
try {
  storage.get(bucketName);
} catch (StorageException e) {
  // surface credential/permission problem before the job starts
}

Try / catch

catch (IOException e) {
  Throwable c = e.getCause();
  if (c instanceof StorageException se) {
    int code = se.getCode();
    boolean transientErr = code == 429 || code >= 500;
    if (transientErr && attempt < MAX) { backoff(attempt++); retryList(); return; }
  }
  throw e;
}

Prevention

When it happens

Trigger: gcs.listDirectory(bucketName, objectNamePrefix) where storage.list fails: nonexistent or just-deleted bucket, service account missing storage.objects.list on the bucket, rate limiting on heavy listing fan-out, or a proxy/VPC-SC boundary blocking googleapis.com.

Common situations: Service-account key lacks objectViewer/legacyBucketReader on the target bucket; bucket deleted by another team mid-job; recursive directory walks triggering per-bucket rate limits; corporate proxies or VPC Service Controls rejecting storage.googleapis.com; DNS failures in the cluster.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a84b96d44f7d7ef2. Report an issue: GitHub.