{"record":{"id":"96fb22a243a45609","repo":"apache/druid","slug":"unable-to-delete-from-container-s-the-followin","errorCode":null,"errorMessage":"Unable to delete from container [%s], the following keys [%s]","messagePattern":"Unable to delete from container \\[(.+?)\\], the following keys \\[(.+?)\\]","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-core/azure-extensions/src/main/java/org/apache/druid/storage/azure/AzureStorage.java","lineNumber":289,"sourceCode":"                                   .map(path -> blobContainerClient.getBlobContainerUrl() + \"/\" + path)\n                                   .collect(Collectors.toList());\n\n    boolean hadException = false;\n    List<List<String>> keysChunks = Lists.partition(blobUris, MAX_MULTI_OBJECT_DELETE_SIZE);\n    for (List<String> chunkOfKeys : keysChunks) {\n      try {\n        LOG.info(\"Removing from container [%s] the following files: [%s]\", containerName, chunkOfKeys);\n\n        // We have to call forEach on the response because this is the only way azure batch will throw an exception on an operation failure.\n        blobBatchClient.deleteBlobs(chunkOfKeys, DeleteSnapshotsOptionType.INCLUDE).forEach(response -> LOG.debug(\n            \"Deleting blob with URL %s completed with status code %d%n\",\n            response.getRequest().getUrl(),\n            response.getStatusCode()\n        ));\n      }\n      catch (BlobStorageException | BlobBatchStorageException e) {\n        hadException = true;\n        LOG.noStackTrace().warn(\n            e,\n            \"Unable to delete from container [%s], the following keys [%s]\",\n            containerName,\n            chunkOfKeys\n        );\n      }\n      catch (Exception e) {\n        hadException = true;\n        LOG.noStackTrace().warn(\n            e,\n            \"Unexpected exception occurred when deleting from container [%s], the following keys [%s]\",\n            containerName,\n            chunkOfKeys\n        );\n      }\n    }\n\n    return !hadException;","sourceCodeStart":271,"sourceCodeEnd":307,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/azure-extensions/src/main/java/org/apache/druid/storage/azure/AzureStorage.java#L271-L307","documentation":"AzureStorage.batchDeleteFiles() deletes blobs in chunks using the Azure Blob Batch client. If the batch call throws BlobStorageException or BlobBatchStorageException, hadException is set (which later causes a StorageException to be thrown) and this warning is logged with the container name and the chunk of keys that failed. The message identifies exactly which keys were in the failed batch.","triggerScenarios":"Calling batchDeleteFiles where the Azure batch delete request returns a non-success status (e.g., blob not found is tolerated, but authorization failures, throttling, or network errors throw) for a chunk of keys.","commonSituations":"Azure throttling (429) when deleting thousands of segments at once; storage account credentials lacking delete permission; transient Azure outages; SAS token expiring mid-deletion.","solutions":["Check Azure storage account throttling/limits and increase retry settings or reduce batch concurrency","Verify credentials/SAS have the delete (and list) permissions on the container","Retry the operation — deletion is idempotent for missing blobs","Inspect logged key chunks to identify whether specific blobs (e.g., locked or immutability-policy-protected) fail repeatedly","Handle the thrown StorageException in the caller after the warning (hadException causes it to be raised at the end)"],"exampleFix":"// before\nstorage.batchDeleteFiles(container, keys); // no retry\n// after: retry with backoff on failure\ntry {\n  storage.batchDeleteFiles(container, keys);\n} catch (StorageException e) {\n  retryWithBackoff(() -> storage.batchDeleteFiles(container, keys), 3);\n}","handlingStrategy":"retry","validationCode":"// check delete permission up front\ntry {\n  storage.simpleDelete(container, probeKey);\n} catch (BlobStorageException e) {\n  throw new IllegalStateException(\"Missing delete permission or throttled: \" + e.getStatusCode());\n}","typeGuard":null,"tryCatchPattern":"try {\n  storage.batchDeleteFiles(container, keys);\n} catch (StorageException e) {\n  retryWithBackoff(() -> storage.batchDeleteFiles(container, keys));\n}","preventionTips":["Grant storage account/SAS delete permissions","Use retry with exponential backoff for 429/5xx","Limit batch delete concurrency to avoid Azure throttling","Retry is safe — blob deletes are idempotent"],"tags":["azure","storage","deletion","batch"],"backgroundTag":"api-error-response","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}