apache/hadoop · critical · IOException

Unable to download to any storage directory

Error message

Unable to download to any storage directory

What it means

Before streaming, Util.receiveFile opens a FileOutputStream on every candidate storage path; each failure is logged as WARN 'Unable to download file <f>' and the directory is reported to StorageErrorReporter. If not a single stream could be opened, the download cannot proceed and this IOException is thrown. The real root cause is always in the preceding WARN stack traces, not in this message.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java:260

              LOG.warn("Overwriting existing file " + f
                  + " with file downloaded from " + url);
            }
            FileOutputStream fos = new FileOutputStream(f);
            outputStreams.add(fos);
            streamPathMap.put(fos, f);
          } catch (IOException ioe) {
            LOG.warn("Unable to download file " + f, ioe);
            // This will be null if we're downloading the fsimage to a file
            // outside of an NNStorage directory.
            if (dstStorage != null &&
                (dstStorage instanceof StorageErrorReporter)) {
              ((StorageErrorReporter)dstStorage).reportErrorOnFile(f);
            }
          }
        }

        if (outputStreams.isEmpty()) {
          throw new IOException(
              "Unable to download to any storage directory");
        }
      }

      byte[] buf = new byte[IO_FILE_BUFFER_SIZE];
      while (num > 0) {
        num = stream.read(buf);
        if (num > 0) {
          received += num;
          for (FileOutputStream fos : outputStreams) {
            fos.write(buf, 0, num);
          }
          if (throttler != null) {
            throttler.throttle(num);
          }
        }
      }
      finishedReceiving = true;

View on GitHub (pinned to 2add963021)

Solutions

1. Read the WARN 'Unable to download file …' lines immediately before this exception — they carry the per-path cause (Permission denied, ENOSPC, ENOENT…)
2. Fix ownership/permissions: chown -R hdfs:hdfs the storage dirs and chmod so the NN user can write
3. Free disk space or remount the volume read-write; recreate missing parent directories
4. Verify the path is a directory and not a regular file occupying the name

Example fix

# before: storage dir owned by root
ls -ld /data/nn/current   # drwxr-xr-x root root

# after
chown -R hdfs:hdfs /data/nn
chmod 755 /data/nn

// Verify every destination is writable before starting the transfer
for (File dir : storageDirs) {
  Path p = dir.toPath();
  if (!Files.isDirectory(p) || !Files.isWritable(p)) {
    throw new IllegalStateException("Storage dir not writable: " + p);
  }
  if (p.getFileSystem().getUsableSpace(p) < minFreeBytes) {
    throw new IllegalStateException("Storage dir low on space: " + p);
  }
}

When it happens

Trigger: Every target path fails FileOutputStream creation: permission denied on the storage dir, parent directory missing, path occupied by a directory, disk full, or read-only filesystem.

Common situations: NameNode/2NN restarted as a different user so dfs.namenode.name.dir entries are no longer writable; storage volume full or read-only after an fs error; storage directory deleted but not recreated; SELinux denial on the name dirs.

Related errors

• The length of the feature flag section was negative at {} by
• Cannot create directory {curDir}
• Content-Length header is not provided by the namenode when t
• File " + url + " received length " + received + " is not of
• Mkdirs failed to create {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8141a6806847d00c. Report an issue: GitHub.