What causes "S3 multi-object delete had error(s):\n key=[ ], code=[ ], message=[ ]" in apache/druid?

Inspect the exception's per-key errors (code/message) to identify failed keys.

apache/druid · error · S3MultiObjectDeleteException

S3 multi-object delete had <n> error(s):\n key=[<key>], cod

Error message

S3 multi-object delete had <n> error(s):\n  key=[<key>], code=[<code>], message=[<message>]

What it means

S3Utils.deleteBucketKeys performs S3 multi-object delete; after exhausting retries some keys still failed, so it throws S3MultiObjectDeleteException carrying per-key error codes and messages. Only the failed keys are retried on subsequent attempts.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java:424

          .delete(Delete.builder().objects(remaining).build())
          .build();
      DeleteObjectsResponse response = retryS3Operation(() -> s3Client.deleteObjects(deleteRequest));
      if (!response.hasErrors()) {
        log.info("Deleted %d files", keysToDelete.size());
        return;
      }
      lastErrors = response.errors();
      remaining = lastErrors.stream()
                            .map(e -> ObjectIdentifier.builder().key(e.key()).build())
                            .collect(Collectors.toList());
      log.warn(
          "S3 multi-object delete had %d error(s) on attempt %d/%d, retrying failed keys only",
          remaining.size(),
          attempt + 1,
          retries + 1
      );
    }
    throw new S3MultiObjectDeleteException(lastErrors);
  }

  /**
   * Uploads a file to S3 if possible. First trying to set ACL to give the bucket owner full control of the file before uploading.
   *
   * @param service    S3 client
   * @param disableAcl true if ACL shouldn't be set for the file
   * @param key        The key under which to store the new object.
   * @param file       The path of the file to upload to Amazon S3.
   */
  static void uploadFileIfPossible(
      ServerSideEncryptingAmazonS3 service,
      boolean disableAcl,
      String bucket,
      String key,
      File file
  )
  {

View on GitHub (pinned to 9b90983fd2)

Solutions

Inspect the exception's per-key errors (code/message) to identify failed keys.
Grant s3:DeleteObject (and DeleteObjectVersion if versioned) on the bucket/prefix.
Check for Object Lock, retention, or deny policies on the failing keys.
Re-run the delete; the failed keys are safely retried individually.
Reduce batch size if S3 throttling caused the failures.

Example fix

// before
// IAM: allow s3:ListBucket only
// after
// IAM: also allow
// {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":"arn:aws:s3:::my-bucket/*"}

Defensive patterns

Strategy: try-catch

Validate before calling

// verify delete permission upfront
s3Client.getBucketPolicy(b -> b.bucket(bucket)); // plus IAM dry-run where available

Try / catch

try { S3Utils.deleteObjectsInPath(...); } catch (S3MultiObjectDeleteException e) { e.getErrors().forEach(err -> log.error("delete failed key=%s code=%s msg=%s", err.key(), err.code(), err.message())); }

Prevention

Grant s3:DeleteObject (and versioned variant) on the prefix
Check Object Lock/retention before bulk deletes
Inspect per-key errors instead of retrying blindly

When it happens

Trigger: Deleting objects in a path (e.g. kill task, dropping a datasource) when some keys fail DeleteObjects permanently: access denied, invalid key names, or objects protected by policies, across all retry attempts.

Common situations: IAM policy allowing List but not s3:DeleteObject; S3 Object Lock/versioning preventing deletion; keys with characters rejected by the API; throttling under large batch deletes.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/d5f312060b77d6ef. Report an issue: GitHub.