apache/hadoop · error · DiskErrorException

dir + " is not a directory!"

Error message

dir + " is not a directory!"

What it means

ReadWriteDiskValidator implements the 'read-write' disk check used through DiskValidatorFactory by the YARN NodeManager (yarn.nodemanager.disk-validator=read-write, applied to local-dirs/log-dirs via DirectoryCollection/LocalDirsHandlerService) and LocalDirAllocator. checkStatus(File dir) first verifies dir.isDirectory(); when the path is missing or is a plain file it increments the failure metric and throws DiskErrorException(dir + " is not a directory!").

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java:50

 * ReadWriteDiskValidator is the class to check a directory by to create a file,
 * write some bytes into it, read back, and verify if they are identical.
 * Read time and write time are recorded and put into an
 * {@link ReadWriteDiskValidatorMetrics}.
 */
public class ReadWriteDiskValidator implements DiskValidator {

  public static final String NAME = "read-write";
  private static final Random RANDOM = new Random();

  @Override
  public void checkStatus(File dir) throws DiskErrorException {
    ReadWriteDiskValidatorMetrics metric =
        ReadWriteDiskValidatorMetrics.getMetric(dir.toString());
    Path tmpFile = null;
    try {
      if (!dir.isDirectory()) {
        metric.diskCheckFailed();
        throw new DiskErrorException(dir + " is not a directory!");
      }

      // check the directory presence and permission.
      DiskChecker.checkDir(dir);

      // create a tmp file under the dir
      tmpFile = Files.createTempFile(dir.toPath(), "test", "tmp");

      // write 16 bytes into the tmp file
      byte[] inputBytes = new byte[16];
      RANDOM.nextBytes(inputBytes);
      long startTime = System.nanoTime();
      Files.write(tmpFile, inputBytes);
      long writeLatency = TimeUnit.MICROSECONDS.convert(
          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
      metric.addWriteFileLatency(writeLatency);

      // read back

View on GitHub (pinned to 2add963021)

Solutions

1. mkdir -p the configured directory and chown it to the daemon's user (e.g. yarn)
2. Correct the local-dirs / log-dirs / mapreduce.cluster.local.dir path in config
3. Verify the mount is present (df, /proc/mounts) and the fstab entry mounts before the daemon starts
4. Restart the NodeManager so DirectoryCollection re-checks and marks the directory healthy

Example fix

# before: path missing or is a file
ls -ld /mnt/yarn/local   # missing or '-rw-r--r--'

# after
sudo mkdir -p /mnt/yarn/local && sudo chown yarn:yarn /mnt/yarn/local

File dir = new File(configuredPath);
if (!dir.isDirectory()) {
  throw new IllegalStateException("configured local dir is missing or not a directory: " + dir);
}

try { diskValidator.checkStatus(dir); } catch (DiskErrorException e) { /* mark dir unhealthy, alert ops with e.getMessage() */ }

When it happens

Trigger: checkStatus on a configured local directory that does not exist or is a regular file; a mount that failed at boot so the configured path no longer resolves; a directory replaced by a file; a container/k8s environment where the path was never created.

Common situations: Typos in yarn.nodemanager.local-dirs / log-dirs or mapreduce.cluster.local.dir; hosts booting before network mounts are up; ops creating a placeholder file where the directory should be; UID/ownership changes making the dir inaccessible as a directory check target.

Related errors

• Error in instantiating YarnClient
• Uri without authority: {uri}
• Wrong FS: {path}, expected: {this.getUri()}
• Couldn't obtain an instance of RawLocalFileSystem.
• Data in file has been corrupted.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0c98159dfb017e07. Report an issue: GitHub.