apache/hadoop · critical · IOException

Inconsistent storage detected, image and edits checkpoint ti

Error message

Inconsistent storage detected, image and edits checkpoint times do not match. image checkpoint time = {}edits checkpoint time = {}

What it means

After selecting the newest image and newest edits across pre-transactional storage directories, the inspector compares the checkpoint times at which each was written. If they differ and the special IMAGE-only + EDITS-only directory combination (which is auto-recovered) does not apply, storage is declared inconsistent and startup aborts. Note the message has a formatting quirk: the two values are concatenated without a separator ('...image checkpoint time = <n>edits checkpoint time = <m>').

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java:174

    if (latestEditsSD == null)
      throw new IOException("Edits file is not found in " + editsDirs);
    
    // Make sure we are loading image and edits from same checkpoint
    if (latestNameCheckpointTime > latestEditsCheckpointTime
        && latestNameSD != latestEditsSD
        && latestNameSD.getStorageDirType() == NameNodeDirType.IMAGE
        && latestEditsSD.getStorageDirType() == NameNodeDirType.EDITS) {
      // This is a rare failure when NN has image-only and edits-only
      // storage directories, and fails right after saving images,
      // in some of the storage directories, but before purging edits.
      // See -NOTE- in saveNamespace().
      LOG.error("This is a rare failure scenario!!!");
      LOG.error("Image checkpoint time " + latestNameCheckpointTime +
                " > edits checkpoint time " + latestEditsCheckpointTime);
      LOG.error("Name-node will treat the image as the latest state of " +
                "the namespace. Old edits will be discarded.");
    } else if (latestNameCheckpointTime != latestEditsCheckpointTime) {
      throw new IOException("Inconsistent storage detected, " +
                      "image and edits checkpoint times do not match. " +
                      "image checkpoint time = " + latestNameCheckpointTime +
                      "edits checkpoint time = " + latestEditsCheckpointTime);
    }

    needToSaveAfterRecovery = doRecovery();
    
    FSImageFile file = new FSImageFile(latestNameSD, 
        NNStorage.getStorageFile(latestNameSD, NameNodeFile.IMAGE),
        HdfsServerConstants.INVALID_TXID);
    LinkedList<FSImageFile> ret = new LinkedList<FSImageFile>();
    ret.add(file);
    return ret;
  }

  @Override
  boolean needToSave() {
    return hasOutOfDateStorageDirs ||

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the fstime/fsimage timestamps in every storage directory to find the mismatched one
2. Make all directories consistent: copy one complete checkpoint set (fsimage, fsimage.md5, fstime, edits) into every name dir
3. Or recover via the SecondaryNameNode: place its merged image where -importCheckpoint expects it and boot
4. As a last resort keep one known-good storage directory, boot once, then let the NN re-create the others

Example fix

# before: dirs hold different checkpoint generations
for d in /dfs/name1 /dfs/name2; do echo $d $(cat $d/current/fstime); done
# /dfs/name1 5000
# /dfs/name2 4000   -> mismatch
# after: sync dir2 from dir1's complete checkpoint set
cp /dfs/name1/current/fsimage* /dfs/name1/current/fstime /dfs/name2/current/

// before starting the NN, compare checkpoint times across all name dirs
for (URI u : FSNamesystem.getNamespaceDirs(conf)) {
  File t = new File(u.getPath(), "current/fstime");
  LOG.info(u + " checkpoint time = "
      + (t.exists() ? new String(Files.readAllBytes(t.toPath())).trim() : "<missing>"));
}

When it happens

Trigger: getLatestImages() with latestNameCheckpointTime != latestEditsCheckpointTime while directories are not the dedicated IMAGE/EDITS split: some name dirs hold fsimage/fstime from a different checkpoint generation than the edits files.

Common situations: A checkpoint crashed halfway through writing across multiple storage dirs; an operator restored only some name directories from backup; files from different checkpoint generations mixed across directories.

Related errors

• Image file is not found in {}
• Edits file is not found in {}
• Unable to delete {}
• Unable to rename {} to {}
• The length of the feature flag section was negative at {} by

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9fbb81271d797f92. Report an issue: GitHub.