apache/hadoop · critical · IOException

Keystore not loaded due to some inconsistency ('%s' and '%s'

Error message

Keystore not loaded due to some inconsistency ('%s' and '%s' should not exist together)!!

What it means

JavaKeyStoreProvider.flush() writes the new keystore to <path>_NEW, renames the old file to <path>_OLD, then moves _NEW into place. If both <path> and <path>_NEW exist at load time, a previous flush was interrupted mid-way, so the provider refuses to load rather than guess which file is authoritative.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java:160

   * @throws IOException If there is a problem reading the password file
   * or a problem reading the keystore.
   */
  private void locateKeystore() throws IOException {
    try {
      password = ProviderUtils.locatePassword(KEYSTORE_PASSWORD_ENV_VAR,
          getConf().get(KEYSTORE_PASSWORD_FILE_KEY));
      if (password == null) {
        password = KEYSTORE_PASSWORD_DEFAULT;
      }
      Path oldPath = constructOldPath(path);
      Path newPath = constructNewPath(path);
      keyStore = KeyStore.getInstance(SCHEME_NAME);
      FsPermission perm = null;
      if (fs.exists(path)) {
        // flush did not proceed to completion
        // _NEW should not exist
        if (fs.exists(newPath)) {
          throw new IOException(
              String.format("Keystore not loaded due to some inconsistency "
              + "('%s' and '%s' should not exist together)!!", path, newPath));
        }
        perm = tryLoadFromPath(path, oldPath);
      } else {
        perm = tryLoadIncompleteFlush(oldPath, newPath);
      }
      // Need to save off permissions in case we need to
      // rewrite the keystore in flush()
      permissions = perm;
    } catch (KeyStoreException e) {
      throw new IOException("Can't create keystore: " + e, e);
    } catch (GeneralSecurityException e) {
      throw new IOException("Can't load keystore " + path + " : " + e , e);
    }
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the main keystore file is intact: keytool -list -keystore <path> -storetype jceks (with the provider password)
2. If the main file lists correctly, delete the stale <path>_NEW leftover file and restart KMS
3. If the main file is damaged, recover using the <path>_OLD backup: verify it, then move it into place and remove the bad files
4. Prevent concurrent writers — only one KMS instance should own the keystore file

Example fix

# before: load fails with "should not exist together"
ls /etc/security/keytoolks.jks*
#   ksm.jks  ksm.jks_NEW

# after: verify main keystore, then drop the torn flush target
keytool -list -keystore ksm.jks -storetype jceks -storepass:file pass.txt
rm ksm.jks_NEW
# restart KMS

// Pre-flight check before pointing KMS at a keystore
FileSystem fs = path.getFileSystem(conf);
if (fs.exists(path) && fs.exists(new Path(path + "_NEW"))) {
  throw new IOException("Torn flush detected: " + path + " and " + path + "_NEW both exist; manual repair required");
}

try {
  KeyProvider kp = KeyProviderFactory.getProviders(uri, conf).get(0);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("should not exist together")) {
    // operational repair path: verify keystore, remove _NEW, restart
  }
}

When it happens

Trigger: A process (KMS) crashed or was hard-killed between writing <path>_NEW and completing the rename sequence; both the original keystore and the _NEW flush target are present on disk at next load.

Common situations: KMS restart after an OOM kill or host crash during key creation/deletion/rollover; disk-full errors during flush; two processes flushing the same keystore file concurrently

Related errors

• Can't load keystore ${path} : ${e}
• Can't create keystore: ${e}
• Can't get key ${versionName} from ${path}
• Can't get algorithm for key ${key} from ${path}
• Can't recover key ${key} from ${path}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/98cf0cb278c1d5dd. Report an issue: GitHub.