What causes "Error loading token state from ${key}" in apache/hadoop?

Identify the token row from the key name in the message.

apache/hadoop · error · IOException

Error loading token state from ${key}

Error message

Error loading token state from ${key}

What it means

In the LevelDB-backed state store, delegation token state is stored as rows keyed by TOKEN_STATE_KEY_PREFIX + sequence number. During load, each value is parsed by loadToken; any IOException is wrapped with the offending database key name. It indicates the token's persisted bytes cannot be deserialized — corruption, truncation, or an incompatible writer version.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerLeveldbStateStoreService.java:182

  private int loadTokens(HistoryServerState state) throws IOException {
    int numTokens = 0;
    LeveldbIterator iter = null;
    try {
      iter = new LeveldbIterator(db);
      iter.seek(bytes(TOKEN_STATE_KEY_PREFIX));
      while (iter.hasNext()) {
        Entry<byte[],byte[]> entry = iter.next();
        String key = asString(entry.getKey());
        if (!key.startsWith(TOKEN_STATE_KEY_PREFIX)) {
          break;
        }
        if (LOG.isDebugEnabled()) {
          LOG.debug("Loading token from " + key);
        }
        try {
          loadToken(state, entry.getValue());
        } catch (IOException e) {
          throw new IOException("Error loading token state from " + key, e);
        }
        ++numTokens;
      }
    } catch (DBException e) {
      throw new IOException(e);
    } finally {
      if (iter != null) {
        iter.close();
      }
    }
    return numTokens;
  }

  private void loadToken(HistoryServerState state, byte[] data)
      throws IOException {
    MRDelegationTokenIdentifier tokenId = new MRDelegationTokenIdentifier();
    long renewDate;
    DataInputStream in = new DataInputStream(new ByteArrayInputStream(data));

View on GitHub (pinned to 2add963021)

Solutions

Identify the token row from the key name in the message.
Confirm version compatibility between the JHS reading and the one that wrote the store.
If the token state can be sacrificed, stop JHS and remove the LevelDB directory; clients re-authenticate with fresh tokens.
Otherwise restore from backup; do not hand-edit LevelDB contents.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  // JHS startup with Leveldb state store
} catch (IOException e) {
  if (e.getMessage().startsWith("Error loading token state")) {
    // corrupt/incompatible token row: purge or restore the leveldb dir
  }
}

Prevention

Avoid hard kills of the JHS process during token updates.
Do not share or manually modify the LevelDB state directory.
Verify version compatibility before pointing JHS at existing state.

When it happens

Trigger: Corrupted LevelDB values from disk faults or unclean shutdown; store written by an incompatible Hadoop version; a partially-written row from a crash during updateToken.

Common situations: Disk issues on the local state directory; downgrades after an upgrade; state directory shared or modified by other processes.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7db8ae2b8bef5dec. Report an issue: GitHub.