{"record":{"id":"9fbb81271d797f92","repo":"apache/hadoop","slug":"inconsistent-storage-detected-image-and-edits-che","errorCode":null,"errorMessage":"Inconsistent storage detected, image and edits checkpoint times do not match. image checkpoint time = {}edits checkpoint time = {}","messagePattern":"Inconsistent storage detected, image and edits checkpoint times do not match\\. image checkpoint time = (.+?)edits checkpoint time = (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java","lineNumber":174,"sourceCode":"    if (latestEditsSD == null)\n      throw new IOException(\"Edits file is not found in \" + editsDirs);\n    \n    // Make sure we are loading image and edits from same checkpoint\n    if (latestNameCheckpointTime > latestEditsCheckpointTime\n        && latestNameSD != latestEditsSD\n        && latestNameSD.getStorageDirType() == NameNodeDirType.IMAGE\n        && latestEditsSD.getStorageDirType() == NameNodeDirType.EDITS) {\n      // This is a rare failure when NN has image-only and edits-only\n      // storage directories, and fails right after saving images,\n      // in some of the storage directories, but before purging edits.\n      // See -NOTE- in saveNamespace().\n      LOG.error(\"This is a rare failure scenario!!!\");\n      LOG.error(\"Image checkpoint time \" + latestNameCheckpointTime +\n                \" > edits checkpoint time \" + latestEditsCheckpointTime);\n      LOG.error(\"Name-node will treat the image as the latest state of \" +\n                \"the namespace. Old edits will be discarded.\");\n    } else if (latestNameCheckpointTime != latestEditsCheckpointTime) {\n      throw new IOException(\"Inconsistent storage detected, \" +\n                      \"image and edits checkpoint times do not match. \" +\n                      \"image checkpoint time = \" + latestNameCheckpointTime +\n                      \"edits checkpoint time = \" + latestEditsCheckpointTime);\n    }\n\n    needToSaveAfterRecovery = doRecovery();\n    \n    FSImageFile file = new FSImageFile(latestNameSD, \n        NNStorage.getStorageFile(latestNameSD, NameNodeFile.IMAGE),\n        HdfsServerConstants.INVALID_TXID);\n    LinkedList<FSImageFile> ret = new LinkedList<FSImageFile>();\n    ret.add(file);\n    return ret;\n  }\n\n  @Override\n  boolean needToSave() {\n    return hasOutOfDateStorageDirs ||","sourceCodeStart":156,"sourceCodeEnd":192,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java#L156-L192","documentation":"After selecting the newest image and newest edits across pre-transactional storage directories, the inspector compares the checkpoint times at which each was written. If they differ and the special IMAGE-only + EDITS-only directory combination (which is auto-recovered) does not apply, storage is declared inconsistent and startup aborts. Note the message has a formatting quirk: the two values are concatenated without a separator ('...image checkpoint time = <n>edits checkpoint time = <m>').","triggerScenarios":"getLatestImages() with latestNameCheckpointTime != latestEditsCheckpointTime while directories are not the dedicated IMAGE/EDITS split: some name dirs hold fsimage/fstime from a different checkpoint generation than the edits files.","commonSituations":"A checkpoint crashed halfway through writing across multiple storage dirs; an operator restored only some name directories from backup; files from different checkpoint generations mixed across directories.","solutions":["Inspect the fstime/fsimage timestamps in every storage directory to find the mismatched one","Make all directories consistent: copy one complete checkpoint set (fsimage, fsimage.md5, fstime, edits) into every name dir","Or recover via the SecondaryNameNode: place its merged image where -importCheckpoint expects it and boot","As a last resort keep one known-good storage directory, boot once, then let the NN re-create the others"],"exampleFix":"# before: dirs hold different checkpoint generations\nfor d in /dfs/name1 /dfs/name2; do echo $d $(cat $d/current/fstime); done\n# /dfs/name1 5000\n# /dfs/name2 4000   -> mismatch\n# after: sync dir2 from dir1's complete checkpoint set\ncp /dfs/name1/current/fsimage* /dfs/name1/current/fstime /dfs/name2/current/","handlingStrategy":"validation","validationCode":"// before starting the NN, compare checkpoint times across all name dirs\nfor (URI u : FSNamesystem.getNamespaceDirs(conf)) {\n  File t = new File(u.getPath(), \"current/fstime\");\n  LOG.info(u + \" checkpoint time = \"\n      + (t.exists() ? new String(Files.readAllBytes(t.toPath())).trim() : \"<missing>\"));\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["After restoring any name dir from backup, always restore the complete checkpoint set (fsimage, md5, fstime) into every dir","Script a pre-start check that fstime matches across all storage directories","Investigate crashed checkpoints immediately instead of restarting into mixed state"],"tags":["hadoop","hdfs","namenode","checkpoint","storage-consistency","legacy-format"],"backgroundTag":"inconsistent-storage-state","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}