apache/hadoop · error · IOException

Listing '%s' failed

Error message

Listing '%s' failed

What it means

The recursive object listing path (listObjectInfo) executes GcsListOperation and converts the resulting Blobs into item infos; any StorageException is wrapped as IOException("Listing '<bucket/object>' failed") with the cause attached. The message names the bucket/prefix that failed, and the nested cause carries the true GCS error (permission, invalid request, quota, transient 5xx).

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java:516

    // TODO: Take delimiter from config
    // TODO: Set specific fields

    checkArgument(
            objectName == null || objectName.endsWith("/"),
            String.format("%s should end with /", objectName));
    try {
      List<Blob> blobs = new GcsListOperation.Builder(bucketName, objectName, storage)
          .forRecursiveListing().build()
          .execute();

      List<GoogleCloudStorageItemInfo> result = new ArrayList<>();
      for (Blob blob : blobs) {
        result.add(createItemInfoForBlob(blob));
      }

      return result;
    } catch (StorageException e) {
      throw new IOException(
          String.format("Listing '%s' failed", BlobId.of(bucketName, objectName)), e);
    }
  }

  void deleteObjects(List<StorageResourceId> fullObjectNames) throws IOException {
    LOG.trace("deleteObjects({})", fullObjectNames);

    if (fullObjectNames.isEmpty()) {
      return;
    }

    // Validate that all the elements represent StorageObjects.
    for (StorageResourceId toDelete : fullObjectNames) {
      checkArgument(
          toDelete.isStorageObject(),
          "Expected full StorageObject names only, got: %s",
          toDelete);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Grant storage.objects.list (+ view access) on the bucket for listing-heavy workloads
2. Inspect the nested StorageException: 5xx/transient → retry with backoff; 403 → permissions; 400 → fix the prefix
3. Narrow the listed prefix or page through subdirectories to reduce per-request load
4. Retry the listing at the job level for transient failures

try {
  items = gcs.listObjectInfo(bucket, prefix);
} catch (IOException e) {
  if (e.getMessage().startsWith("Listing '") && e.getCause() instanceof StorageException) {
    int code = ((StorageException) e.getCause()).getCode();
    if (code / 100 == 5 || code == 429) { /* back off and retry the listing */ }
  }
  throw e;
}

When it happens

Trigger: Recursive listings (distcp input scanning, getInputPaths, directory trees) where storage.objects.list fails on the bucket: missing permission, malformed prefix/delimiter, request-rate throttling, or a transient GCS error.

Common situations: Service account without storage.objects.list on the bucket; very wide prefixes hitting rate limits during large distcp/Spark scans; GCS transient 500s during heavy listing; recursive listing of a bucket the project cannot enumerate.

Related errors

• Error accessing Bucket %s
• Error accessing %s
• copy(%s->%s) failed.
• listing object '%s' failed.
• Object %s already exists.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/928caa113cbabcb4. Report an issue: GitHub.