{"record":{"id":"7db8ae2b8bef5dec","repo":"apache/hadoop","slug":"error-loading-token-state-from-key","errorCode":null,"errorMessage":"Error loading token state from ${key}","messagePattern":"Error loading token state from (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerLeveldbStateStoreService.java","lineNumber":182,"sourceCode":"  private int loadTokens(HistoryServerState state) throws IOException {\n    int numTokens = 0;\n    LeveldbIterator iter = null;\n    try {\n      iter = new LeveldbIterator(db);\n      iter.seek(bytes(TOKEN_STATE_KEY_PREFIX));\n      while (iter.hasNext()) {\n        Entry<byte[],byte[]> entry = iter.next();\n        String key = asString(entry.getKey());\n        if (!key.startsWith(TOKEN_STATE_KEY_PREFIX)) {\n          break;\n        }\n        if (LOG.isDebugEnabled()) {\n          LOG.debug(\"Loading token from \" + key);\n        }\n        try {\n          loadToken(state, entry.getValue());\n        } catch (IOException e) {\n          throw new IOException(\"Error loading token state from \" + key, e);\n        }\n        ++numTokens;\n      }\n    } catch (DBException e) {\n      throw new IOException(e);\n    } finally {\n      if (iter != null) {\n        iter.close();\n      }\n    }\n    return numTokens;\n  }\n\n  private void loadToken(HistoryServerState state, byte[] data)\n      throws IOException {\n    MRDelegationTokenIdentifier tokenId = new MRDelegationTokenIdentifier();\n    long renewDate;\n    DataInputStream in = new DataInputStream(new ByteArrayInputStream(data));","sourceCodeStart":164,"sourceCodeEnd":200,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerLeveldbStateStoreService.java#L164-L200","documentation":"In the LevelDB-backed state store, delegation token state is stored as rows keyed by TOKEN_STATE_KEY_PREFIX + sequence number. During load, each value is parsed by loadToken; any IOException is wrapped with the offending database key name. It indicates the token's persisted bytes cannot be deserialized — corruption, truncation, or an incompatible writer version.","triggerScenarios":"Corrupted LevelDB values from disk faults or unclean shutdown; store written by an incompatible Hadoop version; a partially-written row from a crash during updateToken.","commonSituations":"Disk issues on the local state directory; downgrades after an upgrade; state directory shared or modified by other processes.","solutions":["Identify the token row from the key name in the message.","Confirm version compatibility between the JHS reading and the one that wrote the store.","If the token state can be sacrificed, stop JHS and remove the LevelDB directory; clients re-authenticate with fresh tokens.","Otherwise restore from backup; do not hand-edit LevelDB contents."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  // JHS startup with Leveldb state store\n} catch (IOException e) {\n  if (e.getMessage().startsWith(\"Error loading token state\")) {\n    // corrupt/incompatible token row: purge or restore the leveldb dir\n  }\n}","preventionTips":["Avoid hard kills of the JHS process during token updates.","Do not share or manually modify the LevelDB state directory.","Verify version compatibility before pointing JHS at existing state."],"tags":["mapreduce","job-history-server","leveldb","state-store","data-corruption"],"backgroundTag":"state-store-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}