Unable to delete from container [%s], the following keys [%s

Error message

Unable to delete from container [%s], the following keys [%s]

What it means

AzureStorage.batchDeleteFiles() deletes blobs in chunks using the Azure Blob Batch client. If the batch call throws BlobStorageException or BlobBatchStorageException, hadException is set (which later causes a StorageException to be thrown) and this warning is logged with the container name and the chunk of keys that failed. The message identifies exactly which keys were in the failed batch.

Source

Thrown at extensions-core/azure-extensions/src/main/java/org/apache/druid/storage/azure/AzureStorage.java:289

                                   .map(path -> blobContainerClient.getBlobContainerUrl() + "/" + path)
                                   .collect(Collectors.toList());

    boolean hadException = false;
    List<List<String>> keysChunks = Lists.partition(blobUris, MAX_MULTI_OBJECT_DELETE_SIZE);
    for (List<String> chunkOfKeys : keysChunks) {
      try {
        LOG.info("Removing from container [%s] the following files: [%s]", containerName, chunkOfKeys);

        // We have to call forEach on the response because this is the only way azure batch will throw an exception on an operation failure.
        blobBatchClient.deleteBlobs(chunkOfKeys, DeleteSnapshotsOptionType.INCLUDE).forEach(response -> LOG.debug(
            "Deleting blob with URL %s completed with status code %d%n",
            response.getRequest().getUrl(),
            response.getStatusCode()
        ));
      }
      catch (BlobStorageException | BlobBatchStorageException e) {
        hadException = true;
        LOG.noStackTrace().warn(
            e,
            "Unable to delete from container [%s], the following keys [%s]",
            containerName,
            chunkOfKeys
        );
      }
      catch (Exception e) {
        hadException = true;
        LOG.noStackTrace().warn(
            e,
            "Unexpected exception occurred when deleting from container [%s], the following keys [%s]",
            containerName,
            chunkOfKeys
        );
      }
    }

    return !hadException;

View on GitHub (pinned to 9b90983fd2)

Solutions

Check Azure storage account throttling/limits and increase retry settings or reduce batch concurrency
Verify credentials/SAS have the delete (and list) permissions on the container
Retry the operation — deletion is idempotent for missing blobs
Inspect logged key chunks to identify whether specific blobs (e.g., locked or immutability-policy-protected) fail repeatedly
Handle the thrown StorageException in the caller after the warning (hadException causes it to be raised at the end)

Example fix

// before
storage.batchDeleteFiles(container, keys); // no retry
// after: retry with backoff on failure
try {
  storage.batchDeleteFiles(container, keys);
} catch (StorageException e) {
  retryWithBackoff(() -> storage.batchDeleteFiles(container, keys), 3);
}

Defensive patterns

Strategy: retry

Validate before calling

// check delete permission up front
try {
  storage.simpleDelete(container, probeKey);
} catch (BlobStorageException e) {
  throw new IllegalStateException("Missing delete permission or throttled: " + e.getStatusCode());
}

Try / catch

try {
  storage.batchDeleteFiles(container, keys);
} catch (StorageException e) {
  retryWithBackoff(() -> storage.batchDeleteFiles(container, keys));
}

Prevention

Grant storage account/SAS delete permissions
Use retry with exponential backoff for 429/5xx
Limit batch delete concurrency to avoid Azure throttling
Retry is safe — blob deletes are idempotent

When it happens

Trigger: Calling batchDeleteFiles where the Azure batch delete request returns a non-success status (e.g., blob not found is tolerated, but authorization failures, throttling, or network errors throw) for a chunk of keys.

Common situations: Azure throttling (429) when deleting thousands of segments at once; storage account credentials lacking delete permission; transient Azure outages; SAS token expiring mid-deletion.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/96fb22a243a45609. Report an issue: GitHub.