{"record":{"id":"3e79fb67d60f34ea","repo":"apache/hadoop","slug":"error-loading-token-master-key-from-key","errorCode":null,"errorMessage":"Error loading token master key from ${key}","messagePattern":"Error loading token master key from (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerLeveldbStateStoreService.java","lineNumber":136,"sourceCode":"      throws IOException {\n    int numKeys = 0;\n    LeveldbIterator iter = null;\n    try {\n      iter = new LeveldbIterator(db);\n      iter.seek(bytes(TOKEN_MASTER_KEY_KEY_PREFIX));\n      while (iter.hasNext()) {\n        Entry<byte[],byte[]> entry = iter.next();\n        String key = asString(entry.getKey());\n        if (!key.startsWith(TOKEN_MASTER_KEY_KEY_PREFIX)) {\n          break;\n        }\n        if (LOG.isDebugEnabled()) {\n          LOG.debug(\"Loading master key from \" + key);\n        }\n        try {\n          loadTokenMasterKey(state, entry.getValue());\n        } catch (IOException e) {\n          throw new IOException(\"Error loading token master key from \" + key,\n              e);\n        }\n        ++numKeys;\n      }\n    } catch (DBException e) {\n      throw new IOException(e);\n    } finally {\n      if (iter != null) {\n        iter.close();\n      }\n    }\n    return numKeys;\n  }\n\n  private void loadTokenMasterKey(HistoryServerState state, byte[] data)\n      throws IOException {\n    DelegationKey key = new DelegationKey();\n    DataInputStream in =","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerLeveldbStateStoreService.java#L118-L154","documentation":"In the LevelDB-backed state store, master keys are rows keyed by TOKEN_MASTER_KEY_KEY_PREFIX + keyId. During state load each value is deserialized via loadTokenMasterKey; any IOException from parsing the bytes is wrapped together with the database key name for diagnosis. The usual root cause is corrupted or version-incompatible data in the LevelDB database under mapreduce.jobhistory.recovery.store.leveldb.path.","triggerScenarios":"LevelDB file corruption from disk errors or unclean shutdown; store written by an incompatible Hadoop version; truncated writes; manual modification of the database.","commonSituations":"Hard kill of JHS mid-write; failing local disk; upgrade/downgrade across schema changes; leftover database from experiments.","solutions":["Note the reported key name to identify which master key fails to parse.","Verify the Hadoop version reading the store matches the one that wrote it (see the 'Loaded state version info' log line).","If token state is disposable, stop JHS and delete the LevelDB directory; it is recreated empty and clients re-authenticate.","Restore the directory from a known-good backup rather than editing it partially."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  // JHS startup with Leveldb state store\n} catch (IOException e) {\n  if (e.getMessage().startsWith(\"Error loading token master key\")) {\n    // identify key from message; purge or restore the leveldb dir, then restart\n  }\n}","preventionTips":["Stop JHS cleanly so LevelDB writes complete.","Keep the state directory on reliable local storage.","Back up the LevelDB directory before version changes."],"tags":["mapreduce","job-history-server","leveldb","state-store","data-corruption"],"backgroundTag":"state-store-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}