apache/hadoop · error · IOException

Could not rename ${tmp} to ${tokenPath}

Error message

Could not rename ${tmp} to ${tokenPath}

What it means

updateToken in the filesystem state store implements token renewal without partial reads: it writes new state to a temp file (UPDATE_TMP_FILE_PREFIX + token name), deletes the old token file, then renames the temp file into place. FileSystem.rename returning false (rather than throwing) produces this IOException. Because the old file is deleted before the rename, a failure here can leave the token record missing plus an orphaned temp file, so the store ends the operation in an inconsistent shape.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryServerFileSystemStateStoreService.java:163

    }

    // Files cannot be atomically replaced, therefore we write a temporary
    // update file, remove the original token file, then rename the update
    // file to the token file. During recovery either the token file will be
    // used or if that is missing and an update file is present then the
    // update file is used.
    Path tokenPath = getTokenPath(tokenId);
    Path tmp = new Path(tokenPath.getParent(),
        UPDATE_TMP_FILE_PREFIX + tokenPath.getName());
    writeFile(tmp, buildTokenData(tokenId, renewDate));
    try {
      deleteFile(tokenPath);
    } catch (IOException e) {
      fs.delete(tmp, false);
      throw e;
    }
    if (!fs.rename(tmp, tokenPath)) {
      throw new IOException("Could not rename " + tmp + " to " + tokenPath);
    }
  }

  @Override
  public void removeToken(MRDelegationTokenIdentifier tokenId)
      throws IOException {
    if (LOG.isDebugEnabled()) {
      LOG.debug("Removing token " + tokenId.getSequenceNumber());
    }
    deleteFile(getTokenPath(tokenId));
  }

  @Override
  public void storeTokenMasterKey(DelegationKey key) throws IOException {
    if (LOG.isDebugEnabled()) {
      LOG.debug("Storing master key " + key.getKeyId());
    }

View on GitHub (pinned to 2add963021)

Solutions

  1. Inspect the store directory for the token file and any leftover update temp file (UPDATE_TMP_FILE_PREFIX...) named in the message.
  2. Restore write+execute permission and correct ownership on the store directory for the JHS user.
  3. Fix the underlying FileSystem issue (NameNode health, disk space), then restart JHS so state reloads and tokens re-register.
  4. If an orphaned temp file remains and the token file is gone, remove the temp file too and let JHS re-issue the token.
Defensive patterns

Strategy: try-catch

Validate before calling

# pre-start check: store dir present and writable by the JHS user
hadoop fs -test -d /jhs-state && hadoop fs -test -w /jhs-state || echo 'fix store dir'

Try / catch

try {
  store.updateToken(tokenId, renewDate);
} catch (IOException e) {
  // rename failed after old file delete: check perms, clean orphaned UPDATE_TMP file, re-run
  log.error("token update failed: {}", e.getMessage());
}

Prevention

When it happens

Trigger: Store directory permissions or ownership changed while JHS is running; HDFS NameNode failover or transient degradation making rename return false; the bucket directory deleted concurrently; a file already present at the target path under rename semantics that refuse overwrite.

Common situations: Permission hardening applied mid-flight on HDFS; a cleanup job removing or moving the recovery store directory; disk-full on a local-path store; HA transition of the NameNode hosting the store.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/003c4e3a46e8e24d. Report an issue: GitHub.