{"record":{"id":"98cf0cb278c1d5dd","repo":"apache/hadoop","slug":"keystore-not-loaded-due-to-some-inconsistency-s","errorCode":null,"errorMessage":"Keystore not loaded due to some inconsistency ('%s' and '%s' should not exist together)!!","messagePattern":"Keystore not loaded due to some inconsistency \\('(.+?)' and '(.+?)' should not exist together\\)!!","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java","lineNumber":160,"sourceCode":"   * @throws IOException If there is a problem reading the password file\n   * or a problem reading the keystore.\n   */\n  private void locateKeystore() throws IOException {\n    try {\n      password = ProviderUtils.locatePassword(KEYSTORE_PASSWORD_ENV_VAR,\n          getConf().get(KEYSTORE_PASSWORD_FILE_KEY));\n      if (password == null) {\n        password = KEYSTORE_PASSWORD_DEFAULT;\n      }\n      Path oldPath = constructOldPath(path);\n      Path newPath = constructNewPath(path);\n      keyStore = KeyStore.getInstance(SCHEME_NAME);\n      FsPermission perm = null;\n      if (fs.exists(path)) {\n        // flush did not proceed to completion\n        // _NEW should not exist\n        if (fs.exists(newPath)) {\n          throw new IOException(\n              String.format(\"Keystore not loaded due to some inconsistency \"\n              + \"('%s' and '%s' should not exist together)!!\", path, newPath));\n        }\n        perm = tryLoadFromPath(path, oldPath);\n      } else {\n        perm = tryLoadIncompleteFlush(oldPath, newPath);\n      }\n      // Need to save off permissions in case we need to\n      // rewrite the keystore in flush()\n      permissions = perm;\n    } catch (KeyStoreException e) {\n      throw new IOException(\"Can't create keystore: \" + e, e);\n    } catch (GeneralSecurityException e) {\n      throw new IOException(\"Can't load keystore \" + path + \" : \" + e , e);\n    }\n  }\n\n  /**","sourceCodeStart":142,"sourceCodeEnd":178,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java#L142-L178","documentation":"JavaKeyStoreProvider.flush() writes the new keystore to <path>_NEW, renames the old file to <path>_OLD, then moves _NEW into place. If both <path> and <path>_NEW exist at load time, a previous flush was interrupted mid-way, so the provider refuses to load rather than guess which file is authoritative.","triggerScenarios":"A process (KMS) crashed or was hard-killed between writing <path>_NEW and completing the rename sequence; both the original keystore and the _NEW flush target are present on disk at next load.","commonSituations":"KMS restart after an OOM kill or host crash during key creation/deletion/rollover; disk-full errors during flush; two processes flushing the same keystore file concurrently","solutions":["Verify the main keystore file is intact: keytool -list -keystore <path> -storetype jceks (with the provider password)","If the main file lists correctly, delete the stale <path>_NEW leftover file and restart KMS","If the main file is damaged, recover using the <path>_OLD backup: verify it, then move it into place and remove the bad files","Prevent concurrent writers — only one KMS instance should own the keystore file"],"exampleFix":"# before: load fails with \"should not exist together\"\nls /etc/security/keytoolks.jks*\n#   ksm.jks  ksm.jks_NEW\n\n# after: verify main keystore, then drop the torn flush target\nkeytool -list -keystore ksm.jks -storetype jceks -storepass:file pass.txt\nrm ksm.jks_NEW\n# restart KMS","handlingStrategy":"validation","validationCode":"// Pre-flight check before pointing KMS at a keystore\nFileSystem fs = path.getFileSystem(conf);\nif (fs.exists(path) && fs.exists(new Path(path + \"_NEW\"))) {\n  throw new IOException(\"Torn flush detected: \" + path + \" and \" + path + \"_NEW both exist; manual repair required\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  KeyProvider kp = KeyProviderFactory.getProviders(uri, conf).get(0);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"should not exist together\")) {\n    // operational repair path: verify keystore, remove _NEW, restart\n  }\n}","preventionTips":["Run exactly one KMS writer per keystore file — no concurrent flushes","Stop KMS cleanly (SIGTERM, wait) rather than SIGKILL during key operations","Monitor for *.jks_NEW leftovers as an early warning of torn flushes","Keep filesystem-level backups of the keystore, _OLD and _NEW artifacts included"],"tags":["keystore","kms","crypto","flush","corruption","file-state"],"backgroundTag":"torn-write-inconsistency","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}