apache/hadoop · error · IOException

${tokenPath} already exists

Error message

${tokenPath} already exists

What it means

Thrown by the JobHistoryServer's filesystem-backed recovery state store when storeToken tries to persist a delegation token whose target file (TOKEN_FILE_PREFIX + sequence number, sharded into bucket directories) already exists on the backing FileSystem. The store is only active when mapreduce.jobhistory.recovery.enable=true and the filesystem state store is configured, so this indicates the store already holds state for the same token sequence number. It almost always means a stale, reused, or shared recovery-store directory rather than a code defect.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerFileSystemStateStoreService.java:134

  @Override
  public HistoryServerState loadState() throws IOException {
    LOG.info("Loading history server state from " + rootStatePath);
    HistoryServerState state = new HistoryServerState();
    loadTokenState(state);
    return state;
  }

  @Override
  public void storeToken(MRDelegationTokenIdentifier tokenId,
      Long renewDate) throws IOException {
    if (LOG.isDebugEnabled()) {
      LOG.debug("Storing token " + tokenId.getSequenceNumber());
    }

    Path tokenPath = getTokenPath(tokenId);
    if (fs.exists(tokenPath)) {
      throw new IOException(tokenPath + " already exists");
    }

    createNewFile(tokenPath, buildTokenData(tokenId, renewDate));
  }

  @Override
  public void updateToken(MRDelegationTokenIdentifier tokenId,
      Long renewDate) throws IOException {
    if (LOG.isDebugEnabled()) {
      LOG.debug("Updating token " + tokenId.getSequenceNumber());
    }

    // Files cannot be atomically replaced, therefore we write a temporary
    // update file, remove the original token file, then rename the update
    // file to the token file. During recovery either the token file will be
    // used or if that is missing and an update file is present then the
    // update file is used.
    Path tokenPath = getTokenPath(tokenId);

View on GitHub (pinned to 2add963021)

Solutions

1. Verify exactly one active JobHistoryServer writes to the configured store URI (mapreduce.jobhistory.recovery.store.fs.uri).
2. If persisted token state is disposable, stop the JHS and remove/empty the recovery store directory; clients simply re-authenticate with newly issued tokens.
3. If state must be kept, list the bucket directories, locate the file named token_<sequence> from the message, and remove only that conflicting file.
4. Confirm mapreduce.jobhistory.recovery.enable / mapreduce.jobhistory.recovery.store.class are intentional and not left enabled from a previous test setup.

try {
  store.storeToken(tokenId, renewDate);
} catch (IOException e) {
  if (e.getMessage().endsWith("already exists")) {
    // duplicate token state in the recovery store: remove stale file or treat as idempotent
  } else {
    throw e;
  }
}

When it happens

Trigger: JHS restarts with recovery enabled against a store directory that still contains a token file with the same sequence number; two JobHistoryServer processes pointed at the same mapreduce.jobhistory.recovery.store.fs.uri; token sequence numbering reset (fresh secret manager) while old token files remain in the store.

Common situations: Redeploying or reinstalling JHS without cleaning the recovery store; sharing one HDFS store location between two JHS instances (test/prod overlap); restoring a store from backup that already contains the tokens about to be re-stored.

Related errors

• ${keyPath} already exists
• Could not rename ${tmp} to ${tokenPath}
• Unexpected file in store: ${dir}
• Could not rename ${tmp} to ${file}
• Unable to delete ${file}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3e7be8c80b66a722. Report an issue: GitHub.