apache/hadoop · critical · java.io.IOException

Upgrade failed in {count} storage directory(ies), previously

Error message

Upgrade failed in {count} storage directory(ies), previously logged.

What it means

doUpgrade() ran NNUpgradeUtil.doUpgrade per storage directory and one or more threw IOException. Failing dirs were reported via storage.reportErrorsOnDirectories() (each is logged), and because losing any directory during an upgrade is fatal, the upgrade aborts with this count summary. The per-directory root causes are in the log lines immediately before this exception.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java:514

    // upgrade shared edit storage first
    if (target.isHaEnabled()) {
      editLog.doUpgradeOfSharedLog();
    }
    for (Iterator<StorageDirectory> it = storage.dirIterator(false); it.hasNext();) {
      StorageDirectory sd = it.next();
      try {
        NNUpgradeUtil.doUpgrade(sd, storage);
      } catch (IOException ioe) {
        errorSDs.add(sd);
        continue;
      }
    }
    storage.reportErrorsOnDirectories(errorSDs);
    
    isUpgradeFinalized = false;
    if (!storage.getRemovedStorageDirs().isEmpty()) {
      // during upgrade, it's a fatal error to fail any storage directory
      throw new IOException("Upgrade failed in "
          + storage.getRemovedStorageDirs().size()
          + " storage directory(ies), previously logged.");
    }
  }

  void doRollback(FSNamesystem fsns) throws IOException {
    // Rollback is allowed only if there is 
    // a previous fs states in at least one of the storage directories.
    // Directories that don't have previous state do not rollback
    boolean canRollback = false;
    FSImage prevState = new FSImage(conf);
    try {
      prevState.getStorage().layoutVersion = storage.getServiceLayoutVersion();
      for (Iterator<StorageDirectory> it = storage.dirIterator(false); it.hasNext();) {
        StorageDirectory sd = it.next();
        if (!NNUpgradeUtil.canRollBack(sd, storage, prevState.getStorage(),
            storage.getServiceLayoutVersion())) {
          continue;

View on GitHub (pinned to 2add963021)

Solutions

Read the preceding ERROR log lines to identify the failing directory and root cause.
Fix the cause: free space, repair permissions/ownership, restore the mount.
Restore that directory's metadata from a healthy sibling dir or backup, or drop the dead dir from config if acceptable.
Re-run the upgrade; if state is inconsistent, 'hdfs namenode -rollback' to the previous state and retry.

Example fix

# before: upgrade aborts
hdfs --daemon start namenode -upgrade
# IOException: Upgrade failed in 1 storage directory(ies), previously logged.

# after: fix the dir identified in the preceding ERROR log, then retry
chown -R hdfs:hdfs /dfs/nn2
hdfs --daemon start namenode -upgrade

Defensive patterns

Strategy: try-catch

Validate before calling

for (URI u : nameDirs) {
  Path p = Paths.get(u);
  if (!Files.isWritable(p)) {
    throw new IllegalStateException(u + " not writable");
  }
  if (p.toFile().getUsableSpace() < MIN_FREE_BYTES) {
    throw new IllegalStateException(u + " low on space");
  }
}

Try / catch

try {
  fsImage.doUpgrade(namesystem);
} catch (IOException e) {
  // fatal: collect the per-directory ERROR causes, fix the dirs, then rollback or retry
}

Prevention

Preflight free space and write permissions on every name dir before '-upgrade'.
Prefer few, well-monitored name dirs (or QJM) over many fragile local dirs.
Take full metadata backups immediately before upgrading.

When it happens

Trigger: Disk full or I/O errors while renaming current/ to previous/ and writing the new VERSION; permission or ownership problems on one name dir; one of several disks dead at upgrade time; NFS transient failure mid-upgrade.

Common situations: Multi-directory name dirs with one failing member; quotas filling a volume during the upgrade; ownership drift after user changes.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7719998a41334946. Report an issue: GitHub.