apache/hadoop · error · IOException

Token ${tokenFile} should be in bucket ${tokenBucketId}, fou

Error message

Token ${tokenFile} should be in bucket ${tokenBucketId}, found in bucket ${bucketId}

What it means

While loading persisted state, the filesystem store scans each token bucket directory and recomputes each token's bucket from its sequence number (getBucketId). If the token's computed bucket differs from the directory it was actually found in, this IOException is thrown and startup aborts. The store layout is inconsistent with the token data — effectively detected corruption, usually caused by manual rearrangement or version skew.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerFileSystemStateStoreService.java:335

    byte[] keyData = readFile(keyFile, numKeyFileBytes);
    DataInputStream in =
        new DataInputStream(new ByteArrayInputStream(keyData));
    try {
      key.readFields(in);
    } finally {
      IOUtils.cleanupWithLogger(LOG, in);
    }
    state.tokenMasterKeyState.add(key);
  }

  private void loadTokenFromBucket(int bucketId,
      HistoryServerState state, Path tokenFile, long numTokenFileBytes)
          throws IOException {
    MRDelegationTokenIdentifier token =
        loadToken(state, tokenFile, numTokenFileBytes);
    int tokenBucketId = getBucketId(token);
    if (tokenBucketId != bucketId) {
      throw new IOException("Token " + tokenFile
          + " should be in bucket " + tokenBucketId + ", found in bucket "
          + bucketId);
    }
  }

  private MRDelegationTokenIdentifier loadToken(HistoryServerState state,
      Path tokenFile, long numTokenFileBytes) throws IOException {
    MRDelegationTokenIdentifier tokenId = new MRDelegationTokenIdentifier();
    long renewDate;
    byte[] tokenData = readFile(tokenFile, numTokenFileBytes);
    DataInputStream in =
        new DataInputStream(new ByteArrayInputStream(tokenData));
    try {
      tokenId.readFields(in);
      renewDate = in.readLong();
    } finally {
      IOUtils.cleanupWithLogger(LOG, in);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Stop the JHS before touching the store.
2. Recompute the expected bucket from the sequence number in the token filename and move the file to that bucket directory; or remove the offending token file if its state is disposable.
3. Never hand-copy individual files between bucket directories; copy the store tree as a whole.
4. Read the store with the same (or compatible) Hadoop version that wrote it.

try {
  storeService.start(); // triggers state load
} catch (IOException e) {
  if (e.getMessage().contains("should be in bucket")) {
    // misplaced token file: stop JHS, move/remove the file per the sequence number, restart
  }
}

When it happens

Trigger: Token files moved or copied between bucket directories (manual repair, incorrect restore); a store written by a Hadoop version using a different bucket assignment; hand-edited store tree.

Common situations: Restoring a store from backup with files placed into wrong subdirectories; migrating the store across Hadoop versions; scripts manipulating the tree file-by-file.

Related errors

• Error loading token master key from ${key}
• Error loading token state from ${key}
• ${tokenPath} already exists
• Could not rename ${tmp} to ${tokenPath}
• ${keyPath} already exists

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8ecb1d8ff0d6c688. Report an issue: GitHub.