What causes "Deleting resource %s failed." in apache/hadoop?

For 412 causes: re-list the object and delete by its current generation, or treat as already-deleted

apache/hadoop · error · IOException

Deleting resource %s failed.

Error message

Deleting resource %s failed.

What it means

deleteObjects deletes each object, optionally with generationMatch when the caller pinned a generationId; any StorageException is wrapped as IOException("Deleting resource %s failed.") naming the failing resource with the cause attached. Common nested causes are 412 PreconditionFailed (the object's generation changed since it was listed, so the pinned generation no longer matches) and 403 permission errors.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java:552

    }

    // TODO: Do this concurrently
    // TODO: There is duplication. fix it
    for (StorageResourceId toDelete : fullObjectNames) {
      try {
        LOG.trace("Deleting Object ({})", toDelete);
        if (toDelete.hasGenerationId() && toDelete.getGenerationId() != 0) {
          storage.delete(
              BlobId.of(toDelete.getBucketName(), toDelete.getObjectName()),
              Storage.BlobSourceOption.generationMatch(toDelete.getGenerationId()));
        } else {
          // TODO: Remove delete without generationId
          storage.delete(BlobId.of(toDelete.getBucketName(), toDelete.getObjectName()));

          LOG.trace("Deleting Object without generationId ({})", toDelete);
        }
      } catch (StorageException e) {
        throw new IOException(String.format("Deleting resource %s failed.", toDelete), e);
      }
    }
  }

  List<GoogleCloudStorageItemInfo> listBucketInfo() throws IOException {
    List<Bucket> allBuckets = listBucketsInternal();
    List<GoogleCloudStorageItemInfo> bucketInfos = new ArrayList<>(allBuckets.size());
    for (Bucket bucket : allBuckets) {
      bucketInfos.add(createItemInfoForBucket(new StorageResourceId(bucket.getName()), bucket));
    }
    return bucketInfos;
  }


  private List<Bucket> listBucketsInternal() throws IOException {
    checkNotNull(configuration.getProjectId(), "projectId must not be null");
    List<Bucket> allBuckets = new ArrayList<>();
    try {

View on GitHub (pinned to 2add963021)

Solutions

For 412 causes: re-list the object and delete by its current generation, or treat as already-deleted
Grant roles/storage.objectAdmin (or objects.delete) where deletion is expected
Serialize jobs that mutate the same paths to avoid generation races
Retry transient (5xx) nested causes with backoff

Example fix

// before
// delete pinned to a stale generation -> 412 wrapped as "Deleting resource ... failed."
gcs.deleteObjects(Collections.singletonList(resourceWithOldGeneration));

// after: re-fetch info, then delete current generation
GoogleCloudStorageItemInfo fresh = gcs.getItemInfo(
    new StorageResourceId(resource.getBucketName(), resource.getObjectName()));
if (fresh.exists()) {
  gcs.deleteObjects(Collections.singletonList(
      new StorageResourceId(resource.getBucketName(), resource.getObjectName(),
          fresh.getContentGeneration())));
}

Defensive patterns

Strategy: retry

Try / catch

try {
  gcs.deleteObjects(resources);
} catch (IOException e) {
  if (e.getMessage().startsWith("Deleting resource ")
      && e.getCause() instanceof StorageException) {
    int code = ((StorageException) e.getCause()).getCode();
    if (code == 412) { /* generation changed: re-list and re-delete, or treat as gone */ }
    else if (code / 100 == 5) { /* retry with backoff */ }
  }
  throw e;
}

Prevention

Serialize jobs that mutate the same paths to avoid generation races
Grant objects.delete where cleanup is expected
Treat 412 during commit cleanup as 'already modified' and re-resolve, not as fatal

When it happens

Trigger: Deleting an object concurrently modified between listing and delete (generation mismatch → 412); deleting without storage.objects.delete permission; transient GCS errors during commit-time cleanup of _temporary files.

Common situations: Two jobs writing/deleting the same paths concurrently; committer cleanup racing another cleanup; service account with read-only roles asked to delete; transient 5xx during mass deletion.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/902aa666e0eafe9b. Report an issue: GitHub.