apache/hadoop · error · IOException

Can't recover key ${key} from ${path}

Error message

Can't recover key ${key} from ${path}

What it means

In getKeyVersion(), the keystore threw UnrecoverableKeyException: the supplied keystore password cannot decrypt the requested key entry. Almost always a password mismatch between the provider configuration and the keystore (or per-entry key password).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java:344

  @Override
  public KeyVersion getKeyVersion(String versionName) throws IOException {
    readLock.lock();
    try {
      SecretKeySpec key = null;
      try {
        if (!keyStore.containsAlias(versionName)) {
          return null;
        }
        key = (SecretKeySpec) keyStore.getKey(versionName, password);
      } catch (KeyStoreException e) {
        throw new IOException("Can't get key " + versionName + " from " +
                              path, e);
      } catch (NoSuchAlgorithmException e) {
        throw new IOException("Can't get algorithm for key " + key + " from " +
                              path, e);
      } catch (UnrecoverableKeyException e) {
        throw new IOException("Can't recover key " + key + " from " + path, e);
      }
      return new KeyVersion(getBaseName(versionName), versionName, key.getEncoded());
    } finally {
      readLock.unlock();
    }
  }

  @Override
  public List<String> getKeys() throws IOException {
    readLock.lock();
    try {
      ArrayList<String> list = new ArrayList<String>();
      String alias = null;
      try {
        Enumeration<String> e = keyStore.aliases();
        while (e.hasMoreElements()) {
           alias = e.nextElement();
           // only include the metadata key names in the list of names

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the password: keytool -list -keystore <path> -storetype jceks with the same password file content — if it fails, fix the password file referenced by KEYSTORE_PASSWORD_FILE_KEY
2. Strip trailing newlines/whitespace from the password file content expectations — match exactly how the file is read
3. If the store password was intentionally rotated, re-protect or recreate entries and restart KMS
4. Check the nested cause to confirm 'password incorrect' style errors rather than corruption

Example fix

# before: KMS log: Can't recover key mykey@0 from /keys/ks.jks

# after: confirm what password KMS reads
grep -r KEYSTORE_PASSWORD_FILE_KEY /etc/hadoop/
printf '%s' "$(cat /etc/security/ks.pwd)" | keytool -list \
  -keystore /keys/ks.jks -storetype jceks -storepass:file /etc/security/ks.pwd
# mismatch -> update password file, restart KMS

// Prove the password decrypts before serving traffic
KeyStore ks = KeyStore.getInstance("jceks");
ks.load(Files.newInputStream(ksPath), passwordFromFile(conf));
ks.getKey(someAlias, passwordFromFile(conf)); // throws early if wrong

try {
  KeyVersion kv = provider.getKeyVersion(versionName);
} catch (IOException e) {
  if (e.getCause() instanceof UnrecoverableKeyException) {
    // password mismatch: fix KEYSTORE_PASSWORD_FILE_KEY content, restart KMS
  }
}

When it happens

Trigger: KeyProvider.getKeyVersion() with KEYSTORE_PASSWORD_FILE_KEY pointing at a wrong/stale password file, or a keystore whose store password was rotated without updating the KMS configuration. The key entry exists (alias found) but getKey(alias, password) fails to decrypt it.

Common situations: Keystore password rotated by an admin but KMS config not updated; password file has trailing whitespace/newline issues; different environments (prod vs staging) sharing a keystore with different passwords

Related errors

• Can't load keystore ${path} : ${e}
• Can't recover key for ${name} from keystore ${path}
• Keystore not loaded due to some inconsistency ('%s' and '%s'
• Can't create keystore: ${e}
• Can't get key ${versionName} from ${path}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/eb9aa8c1ba4a47b4. Report an issue: GitHub.