apache/hadoop · error · IOException

Error loading token master key from ${key}

Error message

Error loading token master key from ${key}

What it means

In the LevelDB-backed state store, master keys are rows keyed by TOKEN_MASTER_KEY_KEY_PREFIX + keyId. During state load each value is deserialized via loadTokenMasterKey; any IOException from parsing the bytes is wrapped together with the database key name for diagnosis. The usual root cause is corrupted or version-incompatible data in the LevelDB database under mapreduce.jobhistory.recovery.store.leveldb.path.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerLeveldbStateStoreService.java:136

      throws IOException {
    int numKeys = 0;
    LeveldbIterator iter = null;
    try {
      iter = new LeveldbIterator(db);
      iter.seek(bytes(TOKEN_MASTER_KEY_KEY_PREFIX));
      while (iter.hasNext()) {
        Entry<byte[],byte[]> entry = iter.next();
        String key = asString(entry.getKey());
        if (!key.startsWith(TOKEN_MASTER_KEY_KEY_PREFIX)) {
          break;
        }
        if (LOG.isDebugEnabled()) {
          LOG.debug("Loading master key from " + key);
        }
        try {
          loadTokenMasterKey(state, entry.getValue());
        } catch (IOException e) {
          throw new IOException("Error loading token master key from " + key,
              e);
        }
        ++numKeys;
      }
    } catch (DBException e) {
      throw new IOException(e);
    } finally {
      if (iter != null) {
        iter.close();
      }
    }
    return numKeys;
  }

  private void loadTokenMasterKey(HistoryServerState state, byte[] data)
      throws IOException {
    DelegationKey key = new DelegationKey();
    DataInputStream in =

View on GitHub (pinned to 2add963021)

Solutions

1. Note the reported key name to identify which master key fails to parse.
2. Verify the Hadoop version reading the store matches the one that wrote it (see the 'Loaded state version info' log line).
3. If token state is disposable, stop JHS and delete the LevelDB directory; it is recreated empty and clients re-authenticate.
4. Restore the directory from a known-good backup rather than editing it partially.

try {
  // JHS startup with Leveldb state store
} catch (IOException e) {
  if (e.getMessage().startsWith("Error loading token master key")) {
    // identify key from message; purge or restore the leveldb dir, then restart
  }
}

When it happens

Trigger: LevelDB file corruption from disk errors or unclean shutdown; store written by an incompatible Hadoop version; truncated writes; manual modification of the database.

Common situations: Hard kill of JHS mid-write; failing local disk; upgrade/downgrade across schema changes; leftover database from experiments.

Related errors

• Error loading token state from ${key}
• Token ${tokenFile} should be in bucket ${tokenBucketId}, fou
• ${tokenPath} already exists
• Could not rename ${tmp} to ${tokenPath}
• ${keyPath} already exists

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3e79fb67d60f34ea. Report an issue: GitHub.