{"record":{"id":"31db6d09aad2adbd","repo":"apache/druid","slug":"could-not-delete-user-s-due-to-concurrent-update","errorCode":null,"errorMessage":"Could not delete user[%s] due to concurrent update contention.","messagePattern":"Could not delete user\\[(.+?)\\] due to concurrent update contention\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"extensions-core/druid-basic-security/src/main/java/org/apache/druid/security/basic/authentication/db/updater/CoordinatorBasicAuthenticatorMetadataStorageUpdater.java","lineNumber":319,"sourceCode":"        attempts++;\n      }\n      updateRetryDelay();\n    }\n    throw new ISE(\"Could not create user[%s] due to concurrent update contention.\", userName);\n  }\n\n  private void deleteUserInternal(String prefix, String userName)\n  {\n    int attempts = 0;\n    while (attempts < NUM_RETRIES) {\n      if (deleteUserOnce(prefix, userName)) {\n        return;\n      } else {\n        attempts++;\n      }\n      updateRetryDelay();\n    }\n    throw new ISE(\"Could not delete user[%s] due to concurrent update contention.\", userName);\n  }\n\n  private void updateRetryDelay()\n  {\n    try {\n      Thread.sleep(ThreadLocalRandom.current().nextLong(UPDATE_RETRY_DELAY));\n    }\n    catch (InterruptedException ie) {\n      throw new RuntimeException(ie);\n    }\n  }\n\n  private void setUserCredentialsInternal(String prefix, String userName, BasicAuthenticatorCredentialUpdate update)\n  {\n    BasicAuthenticatorCredentials credentials;\n\n    // use default iteration count from Authenticator if not specified in request\n    if (update.getIterations() == -1) {","sourceCodeStart":301,"sourceCodeEnd":337,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/druid-basic-security/src/main/java/org/apache/druid/security/basic/authentication/db/updater/CoordinatorBasicAuthenticatorMetadataStorageUpdater.java#L301-L337","documentation":"Thrown as an IllegalStateException after the coordinator exhausts all retry attempts (numRetries) trying to delete a user from the metadata store via compare-and-swap. Every CAS attempt failed because another process concurrently modified the user map between read and write. Indicates persistent optimistic-locking contention on the shared user map in metadata storage.","triggerScenarios":"Calling deleteUser (CoordinatorBasicAuthenticatorMetadataStorageUpdater) while other nodes or the coordinator itself repeatedly update the same authenticator user map, so every tryUpdateUserMap CAS within the retry window fails.","commonSituations":"Multiple coordinator leader transitions or several coordinators misconfigured simultaneously; bulk scripted user deletion from different clients; very high user-change API traffic against a single authenticator; slow metadata store (e.g. overloaded MySQL/PostgreSQL metadata store) making each retry window too short.","solutions":["Check that only one coordinator is the leader and extra coordinators are not serving updates","Reduce concurrent user-management API calls against the same authenticator and retry the delete after contention subsides","Verify the metadata store is healthy and responsive; tune coordinator concurrency configs","Increase retry behavior (numRetries / UPDATE_RETRY_DELAY) or upgrade Druid if contention is a known issue"],"exampleFix":"// before\nclient.deleteUser(authenticatorName, userName); // ISE under contention\n// after\ntry {\n  client.deleteUser(authenticatorName, userName);\n} catch (IllegalStateException e) {\n  Thread.sleep(retryBackoffMs);\n  client.deleteUser(authenticatorName, userName);\n}","handlingStrategy":"retry","validationCode":"// before deleting, confirm user exists and quiesce other writers\nResponse r = client.getUser(authenticatorName, userName);\nif (r.getStatus() != 200) { throw new IllegalStateException(\"user missing, nothing to delete\"); }","typeGuard":null,"tryCatchPattern":"try {\n  client.deleteUser(authenticatorName, userName);\n} catch (IllegalStateException e) {\n  // contention: back off and retry a bounded number of times\n  backoffAndRetry(() -> client.deleteUser(authenticatorName, userName));\n}","preventionTips":["Ensure only one coordinator leader performs updates","Serialize bulk user-management operations instead of running them in parallel","Monitor metadata store latency; slow stores shrink the effective retry window"],"tags":["concurrency","optimistic-locking","basic-security","metadata-store"],"backgroundTag":"internal-invariant-violation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}