{"record":{"id":"8b0472c7900a45ac","repo":"apache/druid","slug":"could-not-create-user-s-due-to-concurrent-update","errorCode":null,"errorMessage":"Could not create user[%s] due to concurrent update contention.","messagePattern":"Could not create user\\[(.+?)\\] due to concurrent update contention\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"extensions-core/druid-basic-security/src/main/java/org/apache/druid/security/basic/authentication/db/updater/CoordinatorBasicAuthenticatorMetadataStorageUpdater.java","lineNumber":305,"sourceCode":"  }\n\n  private static String getPrefixedKeyColumn(String keyPrefix, String keyName)\n  {\n    return StringUtils.format(\"basic_authentication_%s_%s\", keyPrefix, keyName);\n  }\n\n  private void createUserInternal(String prefix, String userName)\n  {\n    int attempts = 0;\n    while (attempts < NUM_RETRIES) {\n      if (createUserOnce(prefix, userName)) {\n        return;\n      } else {\n        attempts++;\n      }\n      updateRetryDelay();\n    }\n    throw new ISE(\"Could not create user[%s] due to concurrent update contention.\", userName);\n  }\n\n  private void deleteUserInternal(String prefix, String userName)\n  {\n    int attempts = 0;\n    while (attempts < NUM_RETRIES) {\n      if (deleteUserOnce(prefix, userName)) {\n        return;\n      } else {\n        attempts++;\n      }\n      updateRetryDelay();\n    }\n    throw new ISE(\"Could not delete user[%s] due to concurrent update contention.\", userName);\n  }\n\n  private void updateRetryDelay()\n  {","sourceCodeStart":287,"sourceCodeEnd":323,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/druid-basic-security/src/main/java/org/apache/druid/security/basic/authentication/db/updater/CoordinatorBasicAuthenticatorMetadataStorageUpdater.java#L287-L323","documentation":"Thrown by CoordinatorBasicAuthenticatorMetadataStorageUpdater.createUserInternal after exhausting NUM_RETRIES attempts to create a user without conflicting with concurrent metadata-storage updates. Each attempt re-reads the current user map, applies the create, and writes back; if another writer keeps modifying the map between read and write, contention persists and the method gives up with this ISE. It's a lost-update retry loop, not a lock failure.","triggerScenarios":"Many concurrent POST /users requests to the same authenticator while other config writes are happening; NUM_RETRIES exhausted because read-modify-write CAS keeps failing against the metadata store; slow metadata store amplifying the race window.","commonSituations":"Automated provisioning scripts creating hundreds of users in parallel; multiple coordinator overlord roles or scripts writing authenticator config simultaneously; partition-creators and admins editing users at the same time.","solutions":["Reduce concurrency: serialize user-creation requests (rate-limit the provisioning script or use a single writer)","Retry the createUser call — the failure is contention-based, a later attempt may succeed","Verify only one coordinator leader/writer is active; check for duplicate updater instances","If contention is chronic, batch user creation via a single config update instead of many individual creates"],"exampleFix":"// before\nfor (String u : users) { client.post(\".../users/\" + u); } // parallel calls cause contention\n// after\nusers.forEach(u -> RetryUtils.retry(() -> client.post(\".../users/\" + u), \"createUser\", 5)); // serialized, with backoff","handlingStrategy":"retry","validationCode":"// avoid predictable contention: check existence first\nboolean exists(String user) {\n  byte[] mapBytes = fetchUserMapBytesFromMetadata();\n  Map<String, BasicAuthorizerUser> map = BasicAuthUtils.deserializeAuthorizerUserMap(jsonMapper, mapBytes);\n  return map.containsKey(user);\n}","typeGuard":null,"tryCatchPattern":"try { updater.createUser(authenticatorPrefix, userName); } catch (ISE e) { if (e.getMessage().contains(\"concurrent update contention\")) { backoffAndRetry(userName, 3); } else { throw e; } }","preventionTips":["Serialize user-creation workflows through a single writer or queue","Add client-side exponential backoff around user creation calls","Avoid bulk-creating users in parallel via provisioning scripts","Ensure only one coordinator leader writes authenticator config at a time"],"tags":["concurrency","retry-exhausted","metadata-store"],"backgroundTag":"database-write-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}