apache/hadoop · critical · org.apache.hadoop.hdfs.server.common.InconsistentFSStateException

Directory {dir} is in an inconsistent state: Message digest

Error message

Directory {dir} is in an inconsistent state: Message digest property imageMD5Digest not set for storage directory {root}

What it means

For storage written in the 0.22-era layout (FSIMAGE_CHECKSUM supported, before TXID_BASED_LAYOUT), the fsimage's MD5 is stored not in a sidecar .md5 file but as the imageMD5Digest property in the VERSION file. loadFSImageFile() found that property missing, meaning the storage directory state is inconsistent (hand-edited, partially copied, or damaged VERSION), and throws InconsistentFSStateException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java:818

      FSImageFile imageFile, StartupOption startupOption) throws IOException {
    LOG.info("Planning to load image: " + imageFile);
    StorageDirectory sdForProperties = imageFile.sd;
    storage.readProperties(sdForProperties, startupOption);

    if (NameNodeLayoutVersion.supports(
        LayoutVersion.Feature.TXID_BASED_LAYOUT, getLayoutVersion())) {
      // For txid-based layout, we should have a .md5 file
      // next to the image file
      boolean isRollingRollback = RollingUpgradeStartupOption.ROLLBACK
          .matches(startupOption);
      loadFSImage(imageFile.getFile(), target, recovery, isRollingRollback);
    } else if (NameNodeLayoutVersion.supports(
        LayoutVersion.Feature.FSIMAGE_CHECKSUM, getLayoutVersion())) {
      // In 0.22, we have the checksum stored in the VERSION file.
      String md5 = storage.getDeprecatedProperty(
          NNStorage.DEPRECATED_MESSAGE_DIGEST_PROPERTY);
      if (md5 == null) {
        throw new InconsistentFSStateException(sdForProperties.getRoot(),
            "Message digest property " +
            NNStorage.DEPRECATED_MESSAGE_DIGEST_PROPERTY +
            " not set for storage directory " + sdForProperties.getRoot());
      }
      loadFSImage(imageFile.getFile(), new MD5Hash(md5), target, recovery,
          false);
    } else {
      // We don't have any record of the md5sum
      loadFSImage(imageFile.getFile(), null, target, recovery, false);
    }
  }

  public void initEditLog(StartupOption startOpt) throws IOException {
    Preconditions.checkState(getNamespaceID() != 0,
        "Must know namespace ID before initting edit log");
    String nameserviceId = DFSUtil.getNamenodeNameServiceId(conf);
    if (!HAUtil.isHAEnabled(conf, nameserviceId)) {
      // If this NN is not HA

View on GitHub (pinned to 2add963021)

Solutions

1. Restore the original VERSION file for that storage dir from a backup or sibling directory (all dirs from one namespace share these values).
2. If the fsimage itself is trusted, recompute and append the digest: md5sum fsimage_N, then add the line imageMD5Digest=<value> to VERSION (recovery-only measure).
3. Prefer regenerating a fresh checkpoint from a healthy node over hand-editing VERSION.

Example fix

# before: VERSION (0.22 layout) has no imageMD5Digest line

# after
MD5=$(md5sum /dfs/nn/current/fsimage_* | awk '{print $1}')
echo "imageMD5Digest=$MD5" >> /dfs/nn/current/VERSION

Properties p = new Properties();
try (InputStream is = Files.newInputStream(versionFile)) {
  p.load(is);
}
if (p.getProperty(NNStorage.DEPRECATED_MESSAGE_DIGEST_PROPERTY) == null) {
  // 0.22-era dir with incomplete VERSION — restore or repair before startup
}

When it happens

Trigger: Loading a 0.22-format storage dir whose VERSION lacks imageMD5Digest: VERSION rebuilt by hand, copied incompletely, or restored from the wrong sibling; salvaged disks with partial metadata.

Common situations: Manual metadata surgery on very old clusters; backups that skipped the VERSION file; pre-0.23 namespaces revived for migration.

Related errors

• The length of the feature flag section was negative at {} by
• Found feature flags which we can't handle. Please upgrade yo
• Content-Length header is not provided by the namenode when t
• File " + url + " computed digest " + computedDigest + " does
• Failed to load image from {imageFile}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1cb5f36559715999. Report an issue: GitHub.