apache/hadoop · error · IOException

This namenode has storage info " + myStorageInfoString + " b

Error message

This namenode has storage info " + myStorageInfoString + " but the secondary expected " + theirStorageInfoString

What it means

validateRequest compares the storage-info string the caller sent as a query parameter (namespaceID:clusterID:ctime:blockpoolID of the peer) with the local NameNode storage string; a mismatch means the Secondary/Standby was configured against a different HDFS namespace than the NameNode it is now contacting. The servlet answers 403 and logs the incoming storage info. Which field differs (namespaceID vs clusterID vs blockpoolID) tells you whether it is a different cluster or a partial reformat.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java:254

      String errorMsg = "Only Namenode, Secondary Namenode, and administrators may access "
          + "this servlet";
      sendError(response, HttpServletResponse.SC_FORBIDDEN, errorMsg);
      LOG.warn("Received non-NN/SNN/administrator request for image or edits from "
          + request.getUserPrincipal().getName()
          + " at "
          + request.getRemoteHost());
      throw new IOException(errorMsg);
    }

    String myStorageInfoString = nnImage.getStorage().toColonSeparatedString();
    if (theirStorageInfoString != null
        && !myStorageInfoString.equals(theirStorageInfoString)) {
      String errorMsg = "This namenode has storage info " + myStorageInfoString
          + " but the secondary expected " + theirStorageInfoString;
      sendError(response, HttpServletResponse.SC_FORBIDDEN, errorMsg);
      LOG.warn("Received an invalid request file transfer request "
          + "from a secondary with storage info " + theirStorageInfoString);
      throw new IOException(errorMsg);
    }
  }

  public static void setFileNameHeaders(HttpServletResponse response,
      File file) {
    response.setHeader(CONTENT_DISPOSITION, "attachment; filename=" +
        file.getName());
    response.setHeader(HADOOP_IMAGE_EDITS_HEADER, file.getName());
  }
  
  /**
   * Construct a throttler from conf
   * @param conf configuration
   * @return a data transfer throttler
   */
  public static DataTransferThrottler getThrottler(Configuration conf) {
    long transferBandwidth = conf.getLongBytes(
        DFSConfigKeys.DFS_IMAGE_TRANSFER_RATE_KEY,

View on GitHub (pinned to 2add963021)

Solutions

1. Diff the two storage strings in the message: a different clusterID/blockpoolID means another cluster; a different ctime alone usually means one side was reformatted.
2. For test clusters: reformat both sides together (hdfs namenode -format, then wipe the 2NN checkpoint dir) so namespaceID/clusterID/blockpoolID match.
3. Fix the address configs (dfs.namenode.http-address, dfs.namenode.secondary.http-address / reconf-xml ha addresses) so the secondary talks to the intended active.
4. For real data, restore the stale side from backup instead of reformatting, so the namespace is not destroyed.

Example fix

// before - secondary pointed at the wrong cluster's active
<property>
  <name>dfs.namenode.http-address</name>
  <value>nn.prod-b.example.com:9870</value>
</property>

// after - aligned with the namespace the 2NN checkpoint dirs belong to
<property>
  <name>dfs.namenode.http-address</name>
  <value>nn.prod-a.example.com:9870</value>
</property>

// Before starting the Secondary, compare namespace identities of both sides
static void assertSameNamespace(File nnVersion, File snnVersion) throws IOException {
  Map<String, String> a = parseVersionFile(nnVersion);   // key=value lines
  Map<String, String> b = parseVersionFile(snnVersion);
  for (String k : Arrays.asList("namespaceID", "clusterID", "blockpoolID", "cTime")) {
    if (!Objects.equals(a.get(k), b.get(k))) {
      throw new IllegalStateException("Storage info mismatch on " + k
          + ": nn=" + a.get(k) + " 2nn=" + b.get(k) + " - refusing checkpoint against wrong namespace");
    }
  }
}

try {
  secondary.doCheckpoint();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("but the secondary expected")) {
    LOG.error("2NN checkpoint dirs belong to a different namespace - reformat or repoint, do not retry", e);
    System.exit(1); // config-level failure; retry cannot fix identity mismatch
  }
  throw e;
}

When it happens

Trigger: Secondary started with dfs.namenode.checkpoint.dir (or name dirs) belonging to cluster A while its dfs.namenode.http-address points at the NameNode of cluster B; or the NN namespace was reformatted but the 2NN checkpoint dirs were not (or vice versa).

Common situations: Reusing 2NN checkpoint directories after reformatting the NameNode; IP/DNS changes repointing the secondary at the wrong active; copy-pasted configs between staging and production; HA pairs where one node kept stale storage.

Related errors

• Cannot import image from a checkpoint. "dfs.namenode.checkpo
• Cannot import image from a checkpoint. "dfs.namenode.checkpo
• Can not create a Path from a null string
• Can not create a Path from an empty string
• no SharedFileDescriptorFactory paths were configured.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/584709840ea85c01. Report an issue: GitHub.