apache/druid · error · IllegalStateException

Could not delete user[%s] due to concurrent update contentio

Error message

Could not delete user[%s] due to concurrent update contention.

What it means

Thrown as an IllegalStateException after the coordinator exhausts all retry attempts (numRetries) trying to delete a user from the metadata store via compare-and-swap. Every CAS attempt failed because another process concurrently modified the user map between read and write. Indicates persistent optimistic-locking contention on the shared user map in metadata storage.

Source

Thrown at extensions-core/druid-basic-security/src/main/java/org/apache/druid/security/basic/authentication/db/updater/CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:319

        attempts++;
      }
      updateRetryDelay();
    }
    throw new ISE("Could not create user[%s] due to concurrent update contention.", userName);
  }

  private void deleteUserInternal(String prefix, String userName)
  {
    int attempts = 0;
    while (attempts < NUM_RETRIES) {
      if (deleteUserOnce(prefix, userName)) {
        return;
      } else {
        attempts++;
      }
      updateRetryDelay();
    }
    throw new ISE("Could not delete user[%s] due to concurrent update contention.", userName);
  }

  private void updateRetryDelay()
  {
    try {
      Thread.sleep(ThreadLocalRandom.current().nextLong(UPDATE_RETRY_DELAY));
    }
    catch (InterruptedException ie) {
      throw new RuntimeException(ie);
    }
  }

  private void setUserCredentialsInternal(String prefix, String userName, BasicAuthenticatorCredentialUpdate update)
  {
    BasicAuthenticatorCredentials credentials;

    // use default iteration count from Authenticator if not specified in request
    if (update.getIterations() == -1) {

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Check that only one coordinator is the leader and extra coordinators are not serving updates
  2. Reduce concurrent user-management API calls against the same authenticator and retry the delete after contention subsides
  3. Verify the metadata store is healthy and responsive; tune coordinator concurrency configs
  4. Increase retry behavior (numRetries / UPDATE_RETRY_DELAY) or upgrade Druid if contention is a known issue

Example fix

// before
client.deleteUser(authenticatorName, userName); // ISE under contention
// after
try {
  client.deleteUser(authenticatorName, userName);
} catch (IllegalStateException e) {
  Thread.sleep(retryBackoffMs);
  client.deleteUser(authenticatorName, userName);
}
Defensive patterns

Strategy: retry

Validate before calling

// before deleting, confirm user exists and quiesce other writers
Response r = client.getUser(authenticatorName, userName);
if (r.getStatus() != 200) { throw new IllegalStateException("user missing, nothing to delete"); }

Try / catch

try {
  client.deleteUser(authenticatorName, userName);
} catch (IllegalStateException e) {
  // contention: back off and retry a bounded number of times
  backoffAndRetry(() -> client.deleteUser(authenticatorName, userName));
}

Prevention

When it happens

Trigger: Calling deleteUser (CoordinatorBasicAuthenticatorMetadataStorageUpdater) while other nodes or the coordinator itself repeatedly update the same authenticator user map, so every tryUpdateUserMap CAS within the retry window fails.

Common situations: Multiple coordinator leader transitions or several coordinators misconfigured simultaneously; bulk scripted user deletion from different clients; very high user-change API traffic against a single authenticator; slow metadata store (e.g. overloaded MySQL/PostgreSQL metadata store) making each retry window too short.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/31db6d09aad2adbd. Report an issue: GitHub.