apache/hadoop · error · InvalidEncryptionKeyException

Can't re-compute encryption key for nonce, since the require

Error message

Can't re-compute encryption key for nonce, since the required block key (keyID={keyId}) doesn't exist. Current key: {currentKeyId}

What it means

BlockTokenSecretManager.retrieveDataEncryptionKey throws InvalidEncryptionKeyException when a DataNode asks to re-derive a data encryption key for a (keyId, nonce) pair whose block key is no longer in allKeys. Block keys roll on dfs.block.token.keyUpdateInterval and expired keys are removed after their validity window; once keyId is evicted, the old DataEncryptionKey the DN cached cannot be recomputed. The remedy is built into the protocol: the DN must fetch a fresh DataEncryptionKey from the NameNode.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java:559

        encryptionKey, timer.now() + tokenLifetime,
        encryptionAlgorithm);
  }

  /**
   * Recreate an encryption key based on the given key id and nonce.
   *
   * @param keyId identifier of the secret key used to generate the encryption key.
   * @param nonce random value used to create the encryption key
   * @return the encryption key which corresponds to this (keyId, blockPoolId, nonce)
   * @throws InvalidEncryptionKeyException
   */
  public byte[] retrieveDataEncryptionKey(int keyId, byte[] nonce)
      throws InvalidEncryptionKeyException {
    BlockKey key = null;
    synchronized (this) {
      key = allKeys.get(keyId);
      if (key == null) {
        throw new InvalidEncryptionKeyException("Can't re-compute encryption key"
            + " for nonce, since the required block key (keyID=" + keyId
            + ") doesn't exist. Current key: " + currentKey.getKeyId());
      }
    }
    return createPassword(nonce, key.getKey());
  }

  public BlockKey getCurrentKey() {
    return currentKey;
  }

  @VisibleForTesting
  public synchronized void setKeyUpdateIntervalForTesting(long millis) {
    this.keyUpdateInterval = millis;
  }

  @VisibleForTesting
  public void clearAllKeysForTesting() {

View on GitHub (pinned to 2add963021)

Solutions

1. Handle InvalidEncryptionKeyException on the DN/client side by discarding the cached DataEncryptionKey and calling getBlockTokenSecretManager/NN getDataEncryptionKey() for a fresh one, then retrying the transfer — this is the designed recovery path.
2. Verify key-update configuration so cached keys stay valid: dfs.block.token.keyUpdateInterval and tokenLifetime/maxLifetime must exceed the longest expected transfer.
3. If it fires constantly right after NN restart, check that delegation/block key state (currentKey id continuity) is being persisted/reloaded rather than reset.
4. Fix clock skew between NN and DN (ntp/chrony) so expiry computation agrees on both sides.

Example fix

// before
byte[] ek = btsm.retrieveDataEncryptionKey(keyId, nonce); // InvalidEncryptionKeyException

// after
DataEncryptionKey fresh;
try {
  fresh = btsm.retrieveDataEncryptionKey(keyId, nonce);
} catch (InvalidEncryptionKeyException e) {
  fresh = dnProtocol.getDataEncryptionKey(); // refetch current key from NN
  cachedDEK = fresh; // replace stale cache, retry transfer
}

// Before re-deriving, check the key id is still known
// (allKeys is internal; approximate by comparing against currentKey id window)
if (Math.abs(keyId - btsm.getCurrentKey().getKeyId()) > KEY_WINDOW) {
  keyId = btsm.getCurrentKey().getKeyId(); // will need a fresh DEA anyway
}

try {
  encKey = btsm.retrieveDataEncryptionKey(keyId, nonce);
} catch (InvalidEncryptionKeyException e) {
  // designed recovery: drop cached DEA, fetch fresh key from NN, retry transfer
  DataEncryptionKey dek = namenodeProtocol.getDataEncryptionKey();
  encKey = btsm.retrieveDataEncryptionKey(dek.keyId, dek.nonce);
}

When it happens

Trigger: DataNode calls retrieveDataEncryptionKey(keyId, nonce) via DataNodeProtocol after keyId was evicted from allKeys by key rolling — typically a DN that cached a DEA key longer than the key lifetime, or that was paused/GC'd and resumed with a stale key, while the NN rolled through its key window.

Common situations: Long-lived encrypted data transfers (dfs.encrypt.data.transfer=true) where the DN's cached key outlives the NN key window; NN restart with a new key set that drops old key ids; clock skew making keys expire earlier than the DN expects; heavy load delaying key refetch.

Related errors

• Block pool {bpid} is not found
• currentKey hasn't been initialized.
• Cannot get access token since BlockKeyUpdater is not running
• Got access token error, status message ${message}, ${logInfo
• Could not instantiate KeyProvider for uri: ${providerUri}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d1dcb39ceb396c11. Report an issue: GitHub.