apache/hadoop · error · IOException

Can't recover key for ${name} from keystore ${path}

Error message

Can't recover key for ${name} from keystore ${path}

What it means

In getMetadata(), getKey() threw UnrecoverableKeyException: the configured keystore password cannot decrypt the metadata entry for the named key. The metadata entry is stored as a password-protected key, so a store-password mismatch shows up here rather than at file load.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java:425

      try {
        if (!keyStore.containsAlias(name)) {
          return null;
        }
        Metadata meta = ((KeyMetadata) keyStore.getKey(name, password)).metadata;
        cache.put(name, meta);
        return meta;
      } catch (ClassCastException e) {
        throw new IOException("Can't cast key for " + name + " in keystore " +
            path + " to a KeyMetadata. Key may have been added using " +
            " keytool or some other non-Hadoop method.", e);
      } catch (KeyStoreException e) {
        throw new IOException("Can't get metadata for " + name +
            " from keystore " + path, e);
      } catch (NoSuchAlgorithmException e) {
        throw new IOException("Can't get algorithm for " + name +
            " from keystore " + path, e);
      } catch (UnrecoverableKeyException e) {
        throw new IOException("Can't recover key for " + name +
            " from keystore " + path, e);
      }
    } finally {
      readLock.unlock();
    }
  }

  @Override
  public KeyVersion createKey(String name, byte[] material,
                               Options options) throws IOException {
    Preconditions.checkArgument(name.equals(StringUtils.toLowerCase(name)),
        "Uppercase key names are unsupported: %s", name);
    writeLock.lock();
    try {
      try {
        if (keyStore.containsAlias(name) || cache.containsKey(name)) {
          throw new IOException("Key " + name + " already exists in " + this);
        }

View on GitHub (pinned to 2add963021)

Solutions

1. Verify and fix the password file contents against keytool -list using that exact file
2. Align the password configuration across all KMS hosts and restart
3. If the store password was intentionally changed, re-create entries so metadata is recoverable under the new password

Example fix

# before: hadoop key roll mykey fails: Can't recover key for mykey

# after: validate the exact password KMS uses, then align
printf '%s' "$(cat /etc/security/ks.pwd)" | wc -c   # check for stray newline
keytool -list -keystore /etc/security/ks.jks -storetype jceks \
  -storepass:file /etc/security/ks.pwd
# fix file or keystore until this lists, restart KMS

// Prove metadata entries decrypt before serving key ops
KeyStore ks = KeyStore.getInstance("jceks");
ks.load(in, pwd);
ks.getKey(name + "_metadata", pwd); // UnrecoverableKeyException here = password mismatch

try {
  meta = provider.getMetadata(name);
} catch (IOException e) {
  if (e.getCause() instanceof UnrecoverableKeyException) {
    // store password wrong: align KEYSTORE_PASSWORD_FILE_KEY across hosts, restart
  }
}

When it happens

Trigger: Any key operation that reads metadata (hadoop key list/roll/delete, KMS requests) when the password file (KEYSTORE_PASSWORD_FILE_KEY) does not match the keystore's actual store password, or the entry was protected with a different key password.

Common situations: Password rotated on one host but not all KMS nodes; multiple KMS instances configured with different password files; keystore restored from backup made under another password

Related errors

• Can't load keystore ${path} : ${e}
• Can't recover key ${key} from ${path}
• Can't cast key for ${name} in keystore ${path} to a KeyMetad
• Can't get metadata for ${name} from keystore ${path}
• Keystore not loaded due to some inconsistency ('%s' and '%s'

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e61699f037bff175. Report an issue: GitHub.