apache/druid · error · IllegalStateException

Could not set credentials for user[%s] due to concurrent upd

Error message

Could not set credentials for user[%s] due to concurrent update contention.

What it means

Thrown as an IllegalStateException after exhausting all retry attempts while trying to set credentials for a user in the authenticator metadata store. Every compare-and-swap of the serialized user map failed due to concurrent modification. Indicates the credentials update could not be committed within the retry budget.

Source

Thrown at extensions-core/druid-basic-security/src/main/java/org/apache/druid/security/basic/authentication/db/updater/CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:365

    } else {
      credentials = new BasicAuthenticatorCredentials(update);
    }

    int attempts = 0;
    while (attempts < NUM_RETRIES) {
      if (setUserCredentialOnce(prefix, userName, credentials)) {
        return;
      } else {
        attempts++;
      }
      try {
        Thread.sleep(ThreadLocalRandom.current().nextLong(UPDATE_RETRY_DELAY));
      }
      catch (InterruptedException ie) {
        throw new RuntimeException(ie);
      }
    }
    throw new ISE("Could not set credentials for user[%s] due to concurrent update contention.", userName);
  }

  private boolean createUserOnce(String prefix, String userName)
  {
    byte[] oldValue = getCurrentUserMapBytes(prefix);
    Map<String, BasicAuthenticatorUser> userMap = BasicAuthUtils.deserializeAuthenticatorUserMap(
        objectMapper,
        oldValue
    );
    if (userMap.get(userName) != null) {
      throw new BasicSecurityDBResourceException("User [%s] already exists.", userName);
    } else {
      userMap.put(userName, new BasicAuthenticatorUser(userName, null));
    }
    byte[] newValue = BasicAuthUtils.serializeAuthenticatorUserMap(objectMapper, userMap);
    return tryUpdateUserMap(prefix, userMap, oldValue, newValue);
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Ensure only the elected coordinator performs updates (verify leadership/replicated coordinators config)
  2. Serialize credential updates: retry the setUserCredentials call after the failure
  3. Check metadata store health and latency; reduce concurrent writers
  4. Increase numRetries/UPDATE_RETRY_DELAY in the deployment if contention is frequent

Example fix

// before
client.setUserCredentials(authenticatorName, userName, credentials); // ISE on contention
// after
try {
  client.setUserCredentials(authenticatorName, userName, credentials);
} catch (IllegalStateException e) {
  backoffAndRetry(() -> client.setUserCredentials(authenticatorName, userName, credentials));
}
Defensive patterns

Strategy: retry

Validate before calling

// verify user exists before setting credentials
Response r = client.getUser(authenticatorName, userName);
if (r.getStatus() != 200) { throw new IllegalStateException("create user first"); }

Try / catch

try {
  client.setUserCredentials(authenticatorName, userName, creds);
} catch (IllegalStateException e) {
  backoffAndRetry(() -> client.setUserCredentials(authenticatorName, userName, creds));
}

Prevention

When it happens

Trigger: Calling setUserCredentials (or the startup path from start) for a user while other writers keep changing the same user map so each createUserOnce/setUserCredentialOnce CAS fails across all retries.

Common situations: Parallel credential-rotation scripts hitting multiple coordinators; another admin updating users at the same time; coordinator failover during credential update; metadata store under heavy load delaying writes.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/8980656af0568b49. Report an issue: GitHub.