{"record":{"id":"e3fe2ba39f7738f8","repo":"apache/hadoop","slug":"all-the-storage-failed-while-writing-properties-to","errorCode":null,"errorMessage":"All the storage failed while writing properties to VERSION file","messagePattern":"All the storage failed while writing properties to VERSION file","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java","lineNumber":1184,"sourceCode":"   * @throws IOException When all the storage directory fails to write\n   * VERSION file\n   */\n  @Override\n  public void writeAll() throws IOException {\n    this.layoutVersion = getServiceLayoutVersion();\n    for (StorageDirectory sd : getStorageDirs()) {\n      try {\n        writeProperties(sd);\n      } catch (ClosedByInterruptException e) {\n        LOG.warn(\"Error during write properties to the VERSION file to {}\",\n            sd, e);\n        return;\n      } catch (Exception e) {\n        LOG.warn(\"Error during write properties to the VERSION file to {}\",\n            sd, e);\n        reportErrorsOnDirectory(sd);\n        if (getStorageDirs().isEmpty()) {\n          throw new IOException(\"All the storage failed while writing \" +\n              \"properties to VERSION file\");\n        }\n      }\n    }\n  }\n}","sourceCodeStart":1166,"sourceCodeEnd":1190,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java#L1166-L1190","documentation":"This loop persists properties to every storage directory: each failing dir is logged, removed via reportErrorsOnDirectory, and when getStorageDirs() becomes empty it ends with 'All the storage failed while writing properties to VERSION file'. It runs whenever VERSION must be written (format, upgrade, finalize, saveNamespace bookkeeping); total failure means not a single writable storage directory remains. Note the special case: ClosedByInterruptException is swallowed with only a warn (interrupted shutdown), which is not a storage failure.","triggerScenarios":"Every name/edits dir is unwritable at once - disks full, filesystems remounted read-only after errors, NFS server down, permissions broken - exactly when the NN needs to persist VERSION.","commonSituations":"Cluster-wide disk-full event; central NFS outage when all journals live on it; a permission/SELinux change applied to all dirs; RAID degradation remounting volumes read-only.","solutions":["Restore writability: df -h on every name/edits dir, free space, remount NFS, fix permissions or read-only remounts.","If the log shows ClosedByInterruptException warns instead of failures, this was an interrupted shutdown - no damage; the next successful write persists VERSION.","After restoring, restart the NN so a fresh VERSION write (checkpoint or startup path) confirms health.","Put disk-free and NFS-reachability alerts on every journal directory so the last writable medium never silently dies."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Pre-write check: confirm at least one storage dir is writable\nboolean anyWritable = false;\nfor (String loc : conf.getTrimmedStrings(\"dfs.namenode.name.dir\")) {\n  File d = new File(stripScheme(loc), \"current\");\n  if (d.canWrite() && d.getUsableSpace() > MIN_FREE_BYTES) { anyWritable = true; break; }\n}\nif (!anyWritable) throw new IOException(\"No writable name dir with space - aborting before VERSION write\");","typeGuard":null,"tryCatchPattern":"try {\n  storage.writePropertiesToDirs(); // format/upgrade/finalize paths\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"All the storage failed\")) {\n    // storage-wide outage: fail loudly, page on-call, never continue running\n    throw new IOException(\"Every name/edits dir unwritable - check mounts, space, permissions\", e);\n  }\n  throw e;\n}","preventionTips":["Alert on disk-free and read-only transitions for every name/edits dir.","Avoid a single NFS share as the only journal medium for all dirs.","Treat ClosedByInterruptException warns during shutdown as benign; anything else needs immediate storage triage."],"tags":["hdfs","namenode","storage","disk-full","version-file"],"backgroundTag":"storage-write-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}