apache/hadoop · critical · IOException

Found duplicated storage UUID: %s in %s.

Error message

Found duplicated storage UUID: %s in %s.

What it means

Thrown as IOException from FsDatasetImpl.addVolume during block-pool volume setup when a StorageDirectory's storage UUID is already present in storageMap. Each formatted storage dir gets a unique UUID in its VERSION file; two dirs presenting the same UUID means the DataNode is being pointed at the same storage twice. The message includes the offending UUID and the VERSION file path that carried the duplicate.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:452

      ReplicaMap replicaMap,
      Storage.StorageDirectory sd, StorageType storageType,
      FsVolumeReference ref) throws IOException {
    for (String bp : volumeMap.getBlockPoolList()) {
      lockManager.addLock(LockLevel.VOLUME, bp, ref.getVolume().getStorageID());
      List<String> allSubDirNameForDataSetLock = datasetSubLockStrategy.getAllSubLockNames();
      for (String dir : allSubDirNameForDataSetLock) {
        lockManager.addLock(LockLevel.DIR, bp, ref.getVolume().getStorageID(), dir);
        LOG.info("Added DIR lock for bpid:{}, volume storageid:{}, dir:{}",
            bp, ref.getVolume().getStorageID(), dir);
      }
    }
    DatanodeStorage dnStorage = storageMap.get(sd.getStorageUuid());
    if (dnStorage != null) {
      final String errorMsg = String.format(
          "Found duplicated storage UUID: %s in %s.",
          sd.getStorageUuid(), sd.getVersionFile());
      LOG.error(errorMsg);
      throw new IOException(errorMsg);
    }
    // Check if there is same storage type on the mount.
    // Only useful when same disk tiering is turned on.
    FsVolumeImpl volumeImpl = (FsVolumeImpl) ref.getVolume();
    FsVolumeReference checkRef = volumes
        .getMountVolumeMap()
        .getVolumeRefByMountAndStorageType(
            volumeImpl.getMount(), volumeImpl.getStorageType());
    if (checkRef != null) {
      final String errorMsg = String.format(
          "Storage type %s already exists on same mount: %s.",
          volumeImpl.getStorageType(), volumeImpl.getMount());
      checkRef.close();
      LOG.error(errorMsg);
      throw new IOException(errorMsg);
    }
    volumeMap.mergeAll(replicaMap);
    storageMap.put(sd.getStorageUuid(),

View on GitHub (pinned to 2add963021)

Solutions

1. Read the message: open the named VERSION file, note the storageID, then grep every dir's current/VERSION for the same storageID to find both occurrences.
2. Deduplicate dfs.datanode.data.dir: remove the alias/symlink/duplicate entry so each physical dir appears exactly once.
3. If a dir was cloned, reformat the redundant copy (hdfs datanode -format or delete its VERSION/subdirs) so it gets a fresh UUID.
4. Restart the DataNode and verify each configured dir maps to a distinct UUID.

Example fix

<!-- before: same physical dir via two spellings -->
<property>
  <name>dfs.datanode.data.dir</name>
  <value>/data/dn,[DISK]/data/dn,/mnt/disk1/dn</value>
</property>

<!-- after: one entry per physical directory -->
<property>
  <name>dfs.datanode.data.dir</name>
  <value>/mnt/disk1/dn</value>
</property>

// Before DN start, assert every configured dir has a distinct storage UUID.
Set<String> seen = new HashSet<>();
for (String d : conf.getTrimmedStrings("dfs.datanode.data.dir")) {
  File v = new File(d.replaceAll("^\[[A-Z_]+\]", ""), "current/VERSION");
  if (!v.isFile()) continue; // unformatted dirs get fresh UUIDs
  Properties p = new Properties();
  try (FileInputStream in = new FileInputStream(v)) { p.load(in); }
  String id = p.getProperty("storageID");
  if (id != null && !seen.add(id)) {
    throw new IOException("Duplicate storageID " + id + " at " + v
        + " - deduplicate dfs.datanode.data.dir");
  }
}

When it happens

Trigger: Calling addVolume (DataNode startup / BPOfferService init) with a StorageDirectory whose getVersionFile() carries a storage UUID already registered for that block pool. Typical cause: the same physical directory (or a copy/clone of it) listed twice under dfs.datanode.data.dir, or a bind mount/symlink alias of a dir that is also listed directly.

Common situations: Same dir listed twice in dfs.datanode.data.dir (e.g. file:// and plain path forms); a disk cloned with dd/rsync so both copies share a VERSION UUID; symlinks like /data/dn -> /mnt/disk1 with both paths configured; leftover duplicate entries after config refactoring.

Related errors

• Storage type %s already exists on same mount: %s.
• BlockId {} is not valid.
• No data exists for block {}
• Replica does not exist {}
• Cannot append to a replica with unexpected generation stamp

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a5e6030599f3908a. Report an issue: GitHub.