What causes "Failed to delete %s objects, detail: %s" in apache/hadoop?

Parse the failed key list from the message and retry just those keys with backoff (transient failures usually clear)

apache/hadoop · error · RuntimeException

Failed to delete %s objects, detail: %s

Error message

Failed to delete %s objects, detail: %s

What it means

ObjectUtils chunks keys and calls storage.batchDelete(keys); the TOS backend returns the subset of keys it failed to delete, and any non-empty subset becomes RuntimeException("Failed to delete N objects, detail: k1,k2,...") naming every failed key. It aborts directory deletion or commit cleanup when part of a batch survived.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/ObjectUtils.java:79

    List<String> keysToDelete = Lists.newArrayList();
    for (ObjectInfo obj : objects) {
      keysToDelete.add(obj.key());

      if (keysToDelete.size() == batchSize) {
        batchDelete(storage, keysToDelete);
        keysToDelete.clear();
      }
    }

    if (!keysToDelete.isEmpty()) {
      batchDelete(storage, keysToDelete);
    }
  }

  private static void batchDelete(ObjectStorage storage, List<String> keys) {
    List<String> failedKeys = storage.batchDelete(keys);
    if (!failedKeys.isEmpty()) {
      throw new RuntimeException(String.format("Failed to delete %s objects, detail: %s",
          failedKeys.size(), Joiner.on(",").join(failedKeys)));
    }
  }

  public static Range calculateRange(final long offset, final long limit, final long objSize) {
    Preconditions.checkArgument(offset >= 0,
        String.format("offset is a negative number: %s", offset));
    Preconditions.checkArgument(offset <= objSize,
        String.format("offset: %s is bigger than object size: %s", offset, objSize));
    long len = limit < 0 ? objSize - offset : Math.min(objSize - offset, limit);
    return Range.of(offset, len);
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Parse the failed key list from the message and retry just those keys with backoff (transient failures usually clear)
Verify the credential has tos:DeleteObject on the bucket and the prefixes named in the message
Eliminate concurrent jobs deleting the same tree
If keys were already deleted by someone else, confirm with head() and treat as idempotent success

Example fix

// before
ObjectUtils.bulkDelete(storage, keys);

// after: retry only the failed subset
try {
  ObjectUtils.bulkDelete(storage, keys);
} catch (RuntimeException e) {
  List<String> failed = extractFailedKeys(e.getMessage()); // parse after "detail:"
  for (int attempt = 1; attempt <= 3; attempt++) {
    try {
      ObjectUtils.bulkDelete(storage, failed);
      break;
    } catch (RuntimeException retry) {
      failed = extractFailedKeys(retry.getMessage());
    }
  }
}

Defensive patterns

Strategy: retry

Validate before calling

// pre-verify delete permission and existence only helps partially;
// the real guard is retrying the failed subset reported by the message
List<String> failedKeys = storage.batchDelete(keys);
if (!failedKeys.isEmpty()) {
  LOG.warn("Failed keys, will retry: {}", failedKeys);
}

Try / catch

try {
  ObjectUtils.bulkDelete(storage, keys);
} catch (RuntimeException e) {
  List<String> failed = extractFailedKeys(e.getMessage()); // parse after "detail:"
  // bounded retry with backoff on the failed subset only
  for (int attempt = 1; attempt <= 3 && !failed.isEmpty(); attempt++) {
    failed = storage.batchDelete(failed);
  }
  if (!failed.isEmpty()) {
    throw new IOException("Objects could not be deleted: " + failed);
  }
}

Prevention

Grant tos:DeleteObject on all prefixes the job deletes
Avoid concurrent jobs deleting the same tree
Retry only the failed subset, never the whole batch blindly
Log the failed key list for post-mortem IAM checks

When it happens

Trigger: Bulk deletion where individual objects fail server-side: the IAM credential lacks tos:DeleteObject on some prefixes, a concurrent job already deleted or is deleting the same keys, or transient per-object TOS errors during large recursive deletes.

Common situations: Aborted-commit cleanup racing with another task's cleanup; least-privilege credentials missing DeleteObject; very large directory trees where a few per-key deletes fail.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6e3ae64a14f0b114. Report an issue: GitHub.