apache/druid · error · IllegalStateException

Could not create user[%s] due to concurrent update contentio

Error message

Could not create user[%s] due to concurrent update contention.

What it means

Thrown by CoordinatorBasicAuthenticatorMetadataStorageUpdater.createUserInternal after exhausting NUM_RETRIES attempts to create a user without conflicting with concurrent metadata-storage updates. Each attempt re-reads the current user map, applies the create, and writes back; if another writer keeps modifying the map between read and write, contention persists and the method gives up with this ISE. It's a lost-update retry loop, not a lock failure.

Source

Thrown at extensions-core/druid-basic-security/src/main/java/org/apache/druid/security/basic/authentication/db/updater/CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:305

  }

  private static String getPrefixedKeyColumn(String keyPrefix, String keyName)
  {
    return StringUtils.format("basic_authentication_%s_%s", keyPrefix, keyName);
  }

  private void createUserInternal(String prefix, String userName)
  {
    int attempts = 0;
    while (attempts < NUM_RETRIES) {
      if (createUserOnce(prefix, userName)) {
        return;
      } else {
        attempts++;
      }
      updateRetryDelay();
    }
    throw new ISE("Could not create user[%s] due to concurrent update contention.", userName);
  }

  private void deleteUserInternal(String prefix, String userName)
  {
    int attempts = 0;
    while (attempts < NUM_RETRIES) {
      if (deleteUserOnce(prefix, userName)) {
        return;
      } else {
        attempts++;
      }
      updateRetryDelay();
    }
    throw new ISE("Could not delete user[%s] due to concurrent update contention.", userName);
  }

  private void updateRetryDelay()
  {

View on GitHub (pinned to 9b90983fd2)

Solutions

Reduce concurrency: serialize user-creation requests (rate-limit the provisioning script or use a single writer)
Retry the createUser call — the failure is contention-based, a later attempt may succeed
Verify only one coordinator leader/writer is active; check for duplicate updater instances
If contention is chronic, batch user creation via a single config update instead of many individual creates

Example fix

// before
for (String u : users) { client.post(".../users/" + u); } // parallel calls cause contention
// after
users.forEach(u -> RetryUtils.retry(() -> client.post(".../users/" + u), "createUser", 5)); // serialized, with backoff

Defensive patterns

Strategy: retry

Validate before calling

// avoid predictable contention: check existence first
boolean exists(String user) {
  byte[] mapBytes = fetchUserMapBytesFromMetadata();
  Map<String, BasicAuthorizerUser> map = BasicAuthUtils.deserializeAuthorizerUserMap(jsonMapper, mapBytes);
  return map.containsKey(user);
}

Try / catch

try { updater.createUser(authenticatorPrefix, userName); } catch (ISE e) { if (e.getMessage().contains("concurrent update contention")) { backoffAndRetry(userName, 3); } else { throw e; } }

Prevention

Serialize user-creation workflows through a single writer or queue
Add client-side exponential backoff around user creation calls
Avoid bulk-creating users in parallel via provisioning scripts
Ensure only one coordinator leader writes authenticator config at a time

When it happens

Trigger: Many concurrent POST /users requests to the same authenticator while other config writes are happening; NUM_RETRIES exhausted because read-modify-write CAS keeps failing against the metadata store; slow metadata store amplifying the race window.

Common situations: Automated provisioning scripts creating hundreds of users in parallel; multiple coordinator overlord roles or scripts writing authenticator config simultaneously; partition-creators and admins editing users at the same time.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/8b0472c7900a45ac. Report an issue: GitHub.