apache/hadoop · critical · IOException

All the storage failed while writing properties to VERSION f

Error message

All the storage failed while writing properties to VERSION file

What it means

This loop persists properties to every storage directory: each failing dir is logged, removed via reportErrorsOnDirectory, and when getStorageDirs() becomes empty it ends with 'All the storage failed while writing properties to VERSION file'. It runs whenever VERSION must be written (format, upgrade, finalize, saveNamespace bookkeeping); total failure means not a single writable storage directory remains. Note the special case: ClosedByInterruptException is swallowed with only a warn (interrupted shutdown), which is not a storage failure.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java:1184

   * @throws IOException When all the storage directory fails to write
   * VERSION file
   */
  @Override
  public void writeAll() throws IOException {
    this.layoutVersion = getServiceLayoutVersion();
    for (StorageDirectory sd : getStorageDirs()) {
      try {
        writeProperties(sd);
      } catch (ClosedByInterruptException e) {
        LOG.warn("Error during write properties to the VERSION file to {}",
            sd, e);
        return;
      } catch (Exception e) {
        LOG.warn("Error during write properties to the VERSION file to {}",
            sd, e);
        reportErrorsOnDirectory(sd);
        if (getStorageDirs().isEmpty()) {
          throw new IOException("All the storage failed while writing " +
              "properties to VERSION file");
        }
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Restore writability: df -h on every name/edits dir, free space, remount NFS, fix permissions or read-only remounts.
2. If the log shows ClosedByInterruptException warns instead of failures, this was an interrupted shutdown - no damage; the next successful write persists VERSION.
3. After restoring, restart the NN so a fresh VERSION write (checkpoint or startup path) confirms health.
4. Put disk-free and NFS-reachability alerts on every journal directory so the last writable medium never silently dies.

// Pre-write check: confirm at least one storage dir is writable
boolean anyWritable = false;
for (String loc : conf.getTrimmedStrings("dfs.namenode.name.dir")) {
  File d = new File(stripScheme(loc), "current");
  if (d.canWrite() && d.getUsableSpace() > MIN_FREE_BYTES) { anyWritable = true; break; }
}
if (!anyWritable) throw new IOException("No writable name dir with space - aborting before VERSION write");

try {
  storage.writePropertiesToDirs(); // format/upgrade/finalize paths
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("All the storage failed")) {
    // storage-wide outage: fail loudly, page on-call, never continue running
    throw new IOException("Every name/edits dir unwritable - check mounts, space, permissions", e);
  }
  throw e;
}

When it happens

Trigger: Every name/edits dir is unwritable at once - disks full, filesystems remounted read-only after errors, NFS server down, permissions broken - exactly when the NN needs to persist VERSION.

Common situations: Cluster-wide disk-full event; central NFS outage when all journals live on it; a permission/SELinux change applied to all dirs; RAID degradation remounting volumes read-only.

Related errors

• NameNode directory " + sd.getRoot() + " is not formatted.
• Properties not found for storage directory " + sd
• file VERSION has no block pool Id.
• No storage directories contained VERSION information
• Cannot create directory {curDir}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e3fe2ba39f7738f8. Report an issue: GitHub.