{"record":{"id":"a84b96d44f7d7ef2","repo":"apache/hadoop","slug":"listing-object-s-failed","errorCode":null,"errorMessage":"listing object '%s' failed.","messagePattern":"listing object '(.+?)' failed\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java","lineNumber":911,"sourceCode":"  List<GoogleCloudStorageItemInfo> listDirectory(String bucketName, String objectNamePrefix)\n      throws IOException {\n    checkArgument(\n        objectNamePrefix == null || objectNamePrefix.endsWith(\"/\"),\n        String.format(\"%s should end with /\", objectNamePrefix));\n\n    try {\n      List<Blob> blobs = new GcsListOperation.Builder(bucketName, objectNamePrefix, storage)\n          .forCurrentDirectoryListing().build()\n          .execute();\n\n      ListOperationResult result = new ListOperationResult();\n      for (Blob blob : blobs) {\n        result.add(blob);\n      }\n\n      return result.getItems();\n    } catch (StorageException e) {\n      throw new IOException(\n          String.format(\"listing object '%s' failed.\", BlobId.of(bucketName, objectNamePrefix)),\n          e);\n    }\n  }\n\n  void compose(\n      String bucketName, List<String> sources, String destination, String contentType)\n      throws IOException {\n    LOG.trace(\"compose({}, {}, {}, {})\", bucketName, sources, destination, contentType);\n    List<StorageResourceId> sourceIds =\n        sources.stream()\n            .map(objectName -> new StorageResourceId(bucketName, objectName))\n            .collect(Collectors.toList());\n    StorageResourceId destinationId = new StorageResourceId(bucketName, destination);\n    CreateObjectOptions options =\n        CreateObjectOptions.DEFAULT_OVERWRITE.toBuilder()\n            .setContentType(contentType)\n            .setEnsureEmptyObjectsMetadataMatch(false)","sourceCodeStart":893,"sourceCodeEnd":929,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java#L893-L929","documentation":"GoogleCloudStorage.listDirectory wraps every StorageException from the underlying GcsListOperation (current-directory listing) in an IOException: \"listing object '<bucket, prefix>' failed.\". The real reason is in the nested cause: 404 bucket missing, 401/403 credentials or scopes, 429/5xx quota and transient faults, or network/proxy failures. The prefix must end with '/' or an earlier checkArgument fails with a different exception.","triggerScenarios":"gcs.listDirectory(bucketName, objectNamePrefix) where storage.list fails: nonexistent or just-deleted bucket, service account missing storage.objects.list on the bucket, rate limiting on heavy listing fan-out, or a proxy/VPC-SC boundary blocking googleapis.com.","commonSituations":"Service-account key lacks objectViewer/legacyBucketReader on the target bucket; bucket deleted by another team mid-job; recursive directory walks triggering per-bucket rate limits; corporate proxies or VPC Service Controls rejecting storage.googleapis.com; DNS failures in the cluster.","solutions":["Unwrap and inspect the cause StorageException code: 404 -> bucket gone/typo'd, 401/403 -> fix credentials/scopes, 429/5xx -> retry with backoff.","Verify the bucket exists and the caller has storage.objects.list permission (e.g. `gcloud storage ls gs://bucket` with the same identity).","Retry listing on 429/5xx with exponential backoff and jitter; these are transient by contract.","Check proxy, DNS, and VPC-SC configuration if every storage call fails, not just listings."],"exampleFix":"// before\nList<GoogleCloudStorageItemInfo> items = gcs.listDirectory(bucket, prefix); // IOException, cause hidden\n\n// after\ntry {\n  List<GoogleCloudStorageItemInfo> items = gcs.listDirectory(bucket, prefix);\n} catch (IOException e) {\n  if (e.getCause() instanceof StorageException se) {\n    int code = se.getCode(); // 404, 403, 429, 5xx ...\n  }\n  throw e;\n}","handlingStrategy":"retry","validationCode":"// Cheap pre-flight: confirm the bucket is listable with the same credentials\ntry {\n  storage.get(bucketName);\n} catch (StorageException e) {\n  // surface credential/permission problem before the job starts\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  Throwable c = e.getCause();\n  if (c instanceof StorageException se) {\n    int code = se.getCode();\n    boolean transientErr = code == 429 || code >= 500;\n    if (transientErr && attempt < MAX) { backoff(attempt++); retryList(); return; }\n  }\n  throw e;\n}","preventionTips":["Grant the reader identity storage.objects.list (objectViewer) on every listed bucket.","Bound listing fan-out to stay under per-bucket rate limits.","Always unwrap the StorageException cause before deciding retry vs fail."],"tags":["gcs","listing","storageexception","permissions","ioexception"],"backgroundTag":"cloud-storage-list-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}