{"record":{"id":"6e3ae64a14f0b114","repo":"apache/hadoop","slug":"failed-to-delete-s-objects-detail-s","errorCode":null,"errorMessage":"Failed to delete %s objects, detail: %s","messagePattern":"Failed to delete (.+?) objects, detail: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/ObjectUtils.java","lineNumber":79,"sourceCode":"    List<String> keysToDelete = Lists.newArrayList();\n    for (ObjectInfo obj : objects) {\n      keysToDelete.add(obj.key());\n\n      if (keysToDelete.size() == batchSize) {\n        batchDelete(storage, keysToDelete);\n        keysToDelete.clear();\n      }\n    }\n\n    if (!keysToDelete.isEmpty()) {\n      batchDelete(storage, keysToDelete);\n    }\n  }\n\n  private static void batchDelete(ObjectStorage storage, List<String> keys) {\n    List<String> failedKeys = storage.batchDelete(keys);\n    if (!failedKeys.isEmpty()) {\n      throw new RuntimeException(String.format(\"Failed to delete %s objects, detail: %s\",\n          failedKeys.size(), Joiner.on(\",\").join(failedKeys)));\n    }\n  }\n\n  public static Range calculateRange(final long offset, final long limit, final long objSize) {\n    Preconditions.checkArgument(offset >= 0,\n        String.format(\"offset is a negative number: %s\", offset));\n    Preconditions.checkArgument(offset <= objSize,\n        String.format(\"offset: %s is bigger than object size: %s\", offset, objSize));\n    long len = limit < 0 ? objSize - offset : Math.min(objSize - offset, limit);\n    return Range.of(offset, len);\n  }\n}\n","sourceCodeStart":61,"sourceCodeEnd":93,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/ObjectUtils.java#L61-L93","documentation":"ObjectUtils chunks keys and calls storage.batchDelete(keys); the TOS backend returns the subset of keys it failed to delete, and any non-empty subset becomes RuntimeException(\"Failed to delete N objects, detail: k1,k2,...\") naming every failed key. It aborts directory deletion or commit cleanup when part of a batch survived.","triggerScenarios":"Bulk deletion where individual objects fail server-side: the IAM credential lacks tos:DeleteObject on some prefixes, a concurrent job already deleted or is deleting the same keys, or transient per-object TOS errors during large recursive deletes.","commonSituations":"Aborted-commit cleanup racing with another task's cleanup; least-privilege credentials missing DeleteObject; very large directory trees where a few per-key deletes fail.","solutions":["Parse the failed key list from the message and retry just those keys with backoff (transient failures usually clear)","Verify the credential has tos:DeleteObject on the bucket and the prefixes named in the message","Eliminate concurrent jobs deleting the same tree","If keys were already deleted by someone else, confirm with head() and treat as idempotent success"],"exampleFix":"// before\nObjectUtils.bulkDelete(storage, keys);\n\n// after: retry only the failed subset\ntry {\n  ObjectUtils.bulkDelete(storage, keys);\n} catch (RuntimeException e) {\n  List<String> failed = extractFailedKeys(e.getMessage()); // parse after \"detail:\"\n  for (int attempt = 1; attempt <= 3; attempt++) {\n    try {\n      ObjectUtils.bulkDelete(storage, failed);\n      break;\n    } catch (RuntimeException retry) {\n      failed = extractFailedKeys(retry.getMessage());\n    }\n  }\n}","handlingStrategy":"retry","validationCode":"// pre-verify delete permission and existence only helps partially;\n// the real guard is retrying the failed subset reported by the message\nList<String> failedKeys = storage.batchDelete(keys);\nif (!failedKeys.isEmpty()) {\n  LOG.warn(\"Failed keys, will retry: {}\", failedKeys);\n}","typeGuard":null,"tryCatchPattern":"try {\n  ObjectUtils.bulkDelete(storage, keys);\n} catch (RuntimeException e) {\n  List<String> failed = extractFailedKeys(e.getMessage()); // parse after \"detail:\"\n  // bounded retry with backoff on the failed subset only\n  for (int attempt = 1; attempt <= 3 && !failed.isEmpty(); attempt++) {\n    failed = storage.batchDelete(failed);\n  }\n  if (!failed.isEmpty()) {\n    throw new IOException(\"Objects could not be deleted: \" + failed);\n  }\n}","preventionTips":["Grant tos:DeleteObject on all prefixes the job deletes","Avoid concurrent jobs deleting the same tree","Retry only the failed subset, never the whole batch blindly","Log the failed key list for post-mortem IAM checks"],"tags":["hadoop-tos","object-storage","batch-delete","iam-permissions","cleanup"],"backgroundTag":"batch-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}