{"record":{"id":"902aa666e0eafe9b","repo":"apache/hadoop","slug":"deleting-resource-s-failed","errorCode":null,"errorMessage":"Deleting resource %s failed.","messagePattern":"Deleting resource (.+?) failed\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java","lineNumber":552,"sourceCode":"    }\n\n    // TODO: Do this concurrently\n    // TODO: There is duplication. fix it\n    for (StorageResourceId toDelete : fullObjectNames) {\n      try {\n        LOG.trace(\"Deleting Object ({})\", toDelete);\n        if (toDelete.hasGenerationId() && toDelete.getGenerationId() != 0) {\n          storage.delete(\n              BlobId.of(toDelete.getBucketName(), toDelete.getObjectName()),\n              Storage.BlobSourceOption.generationMatch(toDelete.getGenerationId()));\n        } else {\n          // TODO: Remove delete without generationId\n          storage.delete(BlobId.of(toDelete.getBucketName(), toDelete.getObjectName()));\n\n          LOG.trace(\"Deleting Object without generationId ({})\", toDelete);\n        }\n      } catch (StorageException e) {\n        throw new IOException(String.format(\"Deleting resource %s failed.\", toDelete), e);\n      }\n    }\n  }\n\n  List<GoogleCloudStorageItemInfo> listBucketInfo() throws IOException {\n    List<Bucket> allBuckets = listBucketsInternal();\n    List<GoogleCloudStorageItemInfo> bucketInfos = new ArrayList<>(allBuckets.size());\n    for (Bucket bucket : allBuckets) {\n      bucketInfos.add(createItemInfoForBucket(new StorageResourceId(bucket.getName()), bucket));\n    }\n    return bucketInfos;\n  }\n\n\n  private List<Bucket> listBucketsInternal() throws IOException {\n    checkNotNull(configuration.getProjectId(), \"projectId must not be null\");\n    List<Bucket> allBuckets = new ArrayList<>();\n    try {","sourceCodeStart":534,"sourceCodeEnd":570,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java#L534-L570","documentation":"deleteObjects deletes each object, optionally with generationMatch when the caller pinned a generationId; any StorageException is wrapped as IOException(\"Deleting resource %s failed.\") naming the failing resource with the cause attached. Common nested causes are 412 PreconditionFailed (the object's generation changed since it was listed, so the pinned generation no longer matches) and 403 permission errors.","triggerScenarios":"Deleting an object concurrently modified between listing and delete (generation mismatch → 412); deleting without storage.objects.delete permission; transient GCS errors during commit-time cleanup of _temporary files.","commonSituations":"Two jobs writing/deleting the same paths concurrently; committer cleanup racing another cleanup; service account with read-only roles asked to delete; transient 5xx during mass deletion.","solutions":["For 412 causes: re-list the object and delete by its current generation, or treat as already-deleted","Grant roles/storage.objectAdmin (or objects.delete) where deletion is expected","Serialize jobs that mutate the same paths to avoid generation races","Retry transient (5xx) nested causes with backoff"],"exampleFix":"// before\n// delete pinned to a stale generation -> 412 wrapped as \"Deleting resource ... failed.\"\ngcs.deleteObjects(Collections.singletonList(resourceWithOldGeneration));\n\n// after: re-fetch info, then delete current generation\nGoogleCloudStorageItemInfo fresh = gcs.getItemInfo(\n    new StorageResourceId(resource.getBucketName(), resource.getObjectName()));\nif (fresh.exists()) {\n  gcs.deleteObjects(Collections.singletonList(\n      new StorageResourceId(resource.getBucketName(), resource.getObjectName(),\n          fresh.getContentGeneration())));\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  gcs.deleteObjects(resources);\n} catch (IOException e) {\n  if (e.getMessage().startsWith(\"Deleting resource \")\n      && e.getCause() instanceof StorageException) {\n    int code = ((StorageException) e.getCause()).getCode();\n    if (code == 412) { /* generation changed: re-list and re-delete, or treat as gone */ }\n    else if (code / 100 == 5) { /* retry with backoff */ }\n  }\n  throw e;\n}","preventionTips":["Serialize jobs that mutate the same paths to avoid generation races","Grant objects.delete where cleanup is expected","Treat 412 during commit cleanup as 'already modified' and re-resolve, not as fatal"],"tags":["gcs","hadoop-gcp","delete","generation-mismatch","storageexception"],"backgroundTag":"delete-object-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}