{"record":{"id":"928caa113cbabcb4","repo":"apache/hadoop","slug":"listing-s-failed","errorCode":null,"errorMessage":"Listing '%s' failed","messagePattern":"Listing '(.+?)' failed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java","lineNumber":516,"sourceCode":"    // TODO: Take delimiter from config\n    // TODO: Set specific fields\n\n    checkArgument(\n            objectName == null || objectName.endsWith(\"/\"),\n            String.format(\"%s should end with /\", objectName));\n    try {\n      List<Blob> blobs = new GcsListOperation.Builder(bucketName, objectName, storage)\n          .forRecursiveListing().build()\n          .execute();\n\n      List<GoogleCloudStorageItemInfo> result = new ArrayList<>();\n      for (Blob blob : blobs) {\n        result.add(createItemInfoForBlob(blob));\n      }\n\n      return result;\n    } catch (StorageException e) {\n      throw new IOException(\n          String.format(\"Listing '%s' failed\", BlobId.of(bucketName, objectName)), e);\n    }\n  }\n\n  void deleteObjects(List<StorageResourceId> fullObjectNames) throws IOException {\n    LOG.trace(\"deleteObjects({})\", fullObjectNames);\n\n    if (fullObjectNames.isEmpty()) {\n      return;\n    }\n\n    // Validate that all the elements represent StorageObjects.\n    for (StorageResourceId toDelete : fullObjectNames) {\n      checkArgument(\n          toDelete.isStorageObject(),\n          \"Expected full StorageObject names only, got: %s\",\n          toDelete);\n    }","sourceCodeStart":498,"sourceCodeEnd":534,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java#L498-L534","documentation":"The recursive object listing path (listObjectInfo) executes GcsListOperation and converts the resulting Blobs into item infos; any StorageException is wrapped as IOException(\"Listing '<bucket/object>' failed\") with the cause attached. The message names the bucket/prefix that failed, and the nested cause carries the true GCS error (permission, invalid request, quota, transient 5xx).","triggerScenarios":"Recursive listings (distcp input scanning, getInputPaths, directory trees) where storage.objects.list fails on the bucket: missing permission, malformed prefix/delimiter, request-rate throttling, or a transient GCS error.","commonSituations":"Service account without storage.objects.list on the bucket; very wide prefixes hitting rate limits during large distcp/Spark scans; GCS transient 500s during heavy listing; recursive listing of a bucket the project cannot enumerate.","solutions":["Grant storage.objects.list (+ view access) on the bucket for listing-heavy workloads","Inspect the nested StorageException: 5xx/transient → retry with backoff; 403 → permissions; 400 → fix the prefix","Narrow the listed prefix or page through subdirectories to reduce per-request load","Retry the listing at the job level for transient failures"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  items = gcs.listObjectInfo(bucket, prefix);\n} catch (IOException e) {\n  if (e.getMessage().startsWith(\"Listing '\") && e.getCause() instanceof StorageException) {\n    int code = ((StorageException) e.getCause()).getCode();\n    if (code / 100 == 5 || code == 429) { /* back off and retry the listing */ }\n  }\n  throw e;\n}","preventionTips":["Grant storage.objects.list for listing-heavy jobs","Narrow prefixes to reduce per-request result size and rate pressure","Make listing failure retriable at the job/loop level"],"tags":["gcs","hadoop-gcp","listing","permissions","retry"],"backgroundTag":"list-objects-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}