{"record":{"id":"d1dcb39ceb396c11","repo":"apache/hadoop","slug":"can-t-re-compute-encryption-key-for-nonce-since-t","errorCode":null,"errorMessage":"Can't re-compute encryption key for nonce, since the required block key (keyID={keyId}) doesn't exist. Current key: {currentKeyId}","messagePattern":"Can't re-compute encryption key for nonce, since the required block key \\(keyID=(.+?)\\) doesn't exist\\. Current key: (.+?)","errorType":"exception","errorClass":"InvalidEncryptionKeyException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java","lineNumber":559,"sourceCode":"        encryptionKey, timer.now() + tokenLifetime,\n        encryptionAlgorithm);\n  }\n\n  /**\n   * Recreate an encryption key based on the given key id and nonce.\n   *\n   * @param keyId identifier of the secret key used to generate the encryption key.\n   * @param nonce random value used to create the encryption key\n   * @return the encryption key which corresponds to this (keyId, blockPoolId, nonce)\n   * @throws InvalidEncryptionKeyException\n   */\n  public byte[] retrieveDataEncryptionKey(int keyId, byte[] nonce)\n      throws InvalidEncryptionKeyException {\n    BlockKey key = null;\n    synchronized (this) {\n      key = allKeys.get(keyId);\n      if (key == null) {\n        throw new InvalidEncryptionKeyException(\"Can't re-compute encryption key\"\n            + \" for nonce, since the required block key (keyID=\" + keyId\n            + \") doesn't exist. Current key: \" + currentKey.getKeyId());\n      }\n    }\n    return createPassword(nonce, key.getKey());\n  }\n\n  public BlockKey getCurrentKey() {\n    return currentKey;\n  }\n\n  @VisibleForTesting\n  public synchronized void setKeyUpdateIntervalForTesting(long millis) {\n    this.keyUpdateInterval = millis;\n  }\n\n  @VisibleForTesting\n  public void clearAllKeysForTesting() {","sourceCodeStart":541,"sourceCodeEnd":577,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java#L541-L577","documentation":"BlockTokenSecretManager.retrieveDataEncryptionKey throws InvalidEncryptionKeyException when a DataNode asks to re-derive a data encryption key for a (keyId, nonce) pair whose block key is no longer in allKeys. Block keys roll on dfs.block.token.keyUpdateInterval and expired keys are removed after their validity window; once keyId is evicted, the old DataEncryptionKey the DN cached cannot be recomputed. The remedy is built into the protocol: the DN must fetch a fresh DataEncryptionKey from the NameNode.","triggerScenarios":"DataNode calls retrieveDataEncryptionKey(keyId, nonce) via DataNodeProtocol after keyId was evicted from allKeys by key rolling — typically a DN that cached a DEA key longer than the key lifetime, or that was paused/GC'd and resumed with a stale key, while the NN rolled through its key window.","commonSituations":"Long-lived encrypted data transfers (dfs.encrypt.data.transfer=true) where the DN's cached key outlives the NN key window; NN restart with a new key set that drops old key ids; clock skew making keys expire earlier than the DN expects; heavy load delaying key refetch.","solutions":["Handle InvalidEncryptionKeyException on the DN/client side by discarding the cached DataEncryptionKey and calling getBlockTokenSecretManager/NN getDataEncryptionKey() for a fresh one, then retrying the transfer — this is the designed recovery path.","Verify key-update configuration so cached keys stay valid: dfs.block.token.keyUpdateInterval and tokenLifetime/maxLifetime must exceed the longest expected transfer.","If it fires constantly right after NN restart, check that delegation/block key state (currentKey id continuity) is being persisted/reloaded rather than reset.","Fix clock skew between NN and DN (ntp/chrony) so expiry computation agrees on both sides."],"exampleFix":"// before\nbyte[] ek = btsm.retrieveDataEncryptionKey(keyId, nonce); // InvalidEncryptionKeyException\n\n// after\nDataEncryptionKey fresh;\ntry {\n  fresh = btsm.retrieveDataEncryptionKey(keyId, nonce);\n} catch (InvalidEncryptionKeyException e) {\n  fresh = dnProtocol.getDataEncryptionKey(); // refetch current key from NN\n  cachedDEK = fresh; // replace stale cache, retry transfer\n}","handlingStrategy":"retry","validationCode":"// Before re-deriving, check the key id is still known\n// (allKeys is internal; approximate by comparing against currentKey id window)\nif (Math.abs(keyId - btsm.getCurrentKey().getKeyId()) > KEY_WINDOW) {\n  keyId = btsm.getCurrentKey().getKeyId(); // will need a fresh DEA anyway\n}","typeGuard":null,"tryCatchPattern":"try {\n  encKey = btsm.retrieveDataEncryptionKey(keyId, nonce);\n} catch (InvalidEncryptionKeyException e) {\n  // designed recovery: drop cached DEA, fetch fresh key from NN, retry transfer\n  DataEncryptionKey dek = namenodeProtocol.getDataEncryptionKey();\n  encKey = btsm.retrieveDataEncryptionKey(dek.keyId, dek.nonce);\n}","preventionTips":["Set dfs.block.token.keyUpdateInterval and token lifetimes longer than the longest data transfer.","Cache DataEncryptionKeys only within their expiry; never reuse beyond one key-rolling window.","Keep NN/DN clocks in sync (NTP) so key expiry is computed identically."],"tags":["hdfs","block-token","security","encryption","key-expiry","data-node"],"backgroundTag":"encryption-key-expired","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}