{"record":{"id":"8ecb1d8ff0d6c688","repo":"apache/hadoop","slug":"token-tokenfile-should-be-in-bucket-tokenbuck","errorCode":null,"errorMessage":"Token ${tokenFile} should be in bucket ${tokenBucketId}, found in bucket ${bucketId}","messagePattern":"Token (.+?) should be in bucket (.+?), found in bucket (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerFileSystemStateStoreService.java","lineNumber":335,"sourceCode":"    byte[] keyData = readFile(keyFile, numKeyFileBytes);\n    DataInputStream in =\n        new DataInputStream(new ByteArrayInputStream(keyData));\n    try {\n      key.readFields(in);\n    } finally {\n      IOUtils.cleanupWithLogger(LOG, in);\n    }\n    state.tokenMasterKeyState.add(key);\n  }\n\n  private void loadTokenFromBucket(int bucketId,\n      HistoryServerState state, Path tokenFile, long numTokenFileBytes)\n          throws IOException {\n    MRDelegationTokenIdentifier token =\n        loadToken(state, tokenFile, numTokenFileBytes);\n    int tokenBucketId = getBucketId(token);\n    if (tokenBucketId != bucketId) {\n      throw new IOException(\"Token \" + tokenFile\n          + \" should be in bucket \" + tokenBucketId + \", found in bucket \"\n          + bucketId);\n    }\n  }\n\n  private MRDelegationTokenIdentifier loadToken(HistoryServerState state,\n      Path tokenFile, long numTokenFileBytes) throws IOException {\n    MRDelegationTokenIdentifier tokenId = new MRDelegationTokenIdentifier();\n    long renewDate;\n    byte[] tokenData = readFile(tokenFile, numTokenFileBytes);\n    DataInputStream in =\n        new DataInputStream(new ByteArrayInputStream(tokenData));\n    try {\n      tokenId.readFields(in);\n      renewDate = in.readLong();\n    } finally {\n      IOUtils.cleanupWithLogger(LOG, in);\n    }","sourceCodeStart":317,"sourceCodeEnd":353,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerFileSystemStateStoreService.java#L317-L353","documentation":"While loading persisted state, the filesystem store scans each token bucket directory and recomputes each token's bucket from its sequence number (getBucketId). If the token's computed bucket differs from the directory it was actually found in, this IOException is thrown and startup aborts. The store layout is inconsistent with the token data — effectively detected corruption, usually caused by manual rearrangement or version skew.","triggerScenarios":"Token files moved or copied between bucket directories (manual repair, incorrect restore); a store written by a Hadoop version using a different bucket assignment; hand-edited store tree.","commonSituations":"Restoring a store from backup with files placed into wrong subdirectories; migrating the store across Hadoop versions; scripts manipulating the tree file-by-file.","solutions":["Stop the JHS before touching the store.","Recompute the expected bucket from the sequence number in the token filename and move the file to that bucket directory; or remove the offending token file if its state is disposable.","Never hand-copy individual files between bucket directories; copy the store tree as a whole.","Read the store with the same (or compatible) Hadoop version that wrote it."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  storeService.start(); // triggers state load\n} catch (IOException e) {\n  if (e.getMessage().contains(\"should be in bucket\")) {\n    // misplaced token file: stop JHS, move/remove the file per the sequence number, restart\n  }\n}","preventionTips":["Back up and restore the store tree as a whole, never file-by-file.","Read the store with the same Hadoop version that wrote it.","Never hand-edit bucket directory contents."],"tags":["mapreduce","job-history-server","state-store","data-corruption","startup"],"backgroundTag":"state-store-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}