{"record":{"id":"7719998a41334946","repo":"apache/hadoop","slug":"upgrade-failed-in-count-storage-directory-ies","errorCode":null,"errorMessage":"Upgrade failed in {count} storage directory(ies), previously logged.","messagePattern":"Upgrade failed in (.+?) storage directory\\(ies\\), previously logged\\.","errorType":"exception","errorClass":"java.io.IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java","lineNumber":514,"sourceCode":"    // upgrade shared edit storage first\n    if (target.isHaEnabled()) {\n      editLog.doUpgradeOfSharedLog();\n    }\n    for (Iterator<StorageDirectory> it = storage.dirIterator(false); it.hasNext();) {\n      StorageDirectory sd = it.next();\n      try {\n        NNUpgradeUtil.doUpgrade(sd, storage);\n      } catch (IOException ioe) {\n        errorSDs.add(sd);\n        continue;\n      }\n    }\n    storage.reportErrorsOnDirectories(errorSDs);\n    \n    isUpgradeFinalized = false;\n    if (!storage.getRemovedStorageDirs().isEmpty()) {\n      // during upgrade, it's a fatal error to fail any storage directory\n      throw new IOException(\"Upgrade failed in \"\n          + storage.getRemovedStorageDirs().size()\n          + \" storage directory(ies), previously logged.\");\n    }\n  }\n\n  void doRollback(FSNamesystem fsns) throws IOException {\n    // Rollback is allowed only if there is \n    // a previous fs states in at least one of the storage directories.\n    // Directories that don't have previous state do not rollback\n    boolean canRollback = false;\n    FSImage prevState = new FSImage(conf);\n    try {\n      prevState.getStorage().layoutVersion = storage.getServiceLayoutVersion();\n      for (Iterator<StorageDirectory> it = storage.dirIterator(false); it.hasNext();) {\n        StorageDirectory sd = it.next();\n        if (!NNUpgradeUtil.canRollBack(sd, storage, prevState.getStorage(),\n            storage.getServiceLayoutVersion())) {\n          continue;","sourceCodeStart":496,"sourceCodeEnd":532,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java#L496-L532","documentation":"doUpgrade() ran NNUpgradeUtil.doUpgrade per storage directory and one or more threw IOException. Failing dirs were reported via storage.reportErrorsOnDirectories() (each is logged), and because losing any directory during an upgrade is fatal, the upgrade aborts with this count summary. The per-directory root causes are in the log lines immediately before this exception.","triggerScenarios":"Disk full or I/O errors while renaming current/ to previous/ and writing the new VERSION; permission or ownership problems on one name dir; one of several disks dead at upgrade time; NFS transient failure mid-upgrade.","commonSituations":"Multi-directory name dirs with one failing member; quotas filling a volume during the upgrade; ownership drift after user changes.","solutions":["Read the preceding ERROR log lines to identify the failing directory and root cause.","Fix the cause: free space, repair permissions/ownership, restore the mount.","Restore that directory's metadata from a healthy sibling dir or backup, or drop the dead dir from config if acceptable.","Re-run the upgrade; if state is inconsistent, 'hdfs namenode -rollback' to the previous state and retry."],"exampleFix":"# before: upgrade aborts\nhdfs --daemon start namenode -upgrade\n# IOException: Upgrade failed in 1 storage directory(ies), previously logged.\n\n# after: fix the dir identified in the preceding ERROR log, then retry\nchown -R hdfs:hdfs /dfs/nn2\nhdfs --daemon start namenode -upgrade","handlingStrategy":"try-catch","validationCode":"for (URI u : nameDirs) {\n  Path p = Paths.get(u);\n  if (!Files.isWritable(p)) {\n    throw new IllegalStateException(u + \" not writable\");\n  }\n  if (p.toFile().getUsableSpace() < MIN_FREE_BYTES) {\n    throw new IllegalStateException(u + \" low on space\");\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  fsImage.doUpgrade(namesystem);\n} catch (IOException e) {\n  // fatal: collect the per-directory ERROR causes, fix the dirs, then rollback or retry\n}","preventionTips":["Preflight free space and write permissions on every name dir before '-upgrade'.","Prefer few, well-monitored name dirs (or QJM) over many fragile local dirs.","Take full metadata backups immediately before upgrading."],"tags":["hdfs","namenode","upgrade","io-error","storage-directory"],"backgroundTag":"upgrade-state-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}