apache/hadoop · error · IOException

src.toString() + ": No such file or directory"

Error message

src.toString() + ": No such file or directory"

What it means

Thrown by FileUtil.copy(File src, FileSystem dstFS, Path dst, ...) when the local source path is neither a directory (isDirectory()) nor a regular file (isFile()) but File.canRead() still returns true. The fall-through else branch mimics the shell error '<path>: No such file or directory'. In practice it fires for special files (FIFOs, sockets, device nodes like /dev/null) or a source deleted/raced between the isFile() check and canRead() check. Note the sibling branch: a truly missing file usually fails canRead() first and reports 'Permission denied' instead, so this specific message is the rarer race/special-file case.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java:549

      }
    } else if (src.isFile()) {
      InputStream in = null;
      OutputStream out =null;
      try {
        in = Files.newInputStream(src.toPath());
        out = dstFS.create(dst);
        IOUtils.copyBytes(in, out, conf);
      } catch (IOException e) {
        IOUtils.closeStream( out );
        IOUtils.closeStream( in );
        throw e;
      }
    } else if (!src.canRead()) {
      throw new IOException(src.toString() +
                            ": Permission denied");

    } else {
      throw new IOException(src.toString() +
                            ": No such file or directory");
    }
    if (deleteSource) {
      return FileUtil.fullyDelete(src);
    } else {
      return true;
    }
  }

  /**
   * Copy FileSystem files to local files.
   *
   * @param srcFS srcFs.
   * @param src src.
   * @param dst dst.
   * @param deleteSource delete source.
   * @param conf configuration.
   * @throws IOException raised on errors performing I/O.

View on GitHub (pinned to 2add963021)

Solutions

  1. Pre-check the source before copying: require src.exists() && (src.isFile() || src.isDirectory()) and fail fast with your own message
  2. For directories being copied while mutated, snapshot or lock the tree first (copy from a stable location) to avoid the isFile/canRead race
  3. Exclude or special-case non-regular files (FIFOs, devices) instead of passing them to FileUtil.copy
  4. If the source should exist, verify the path spelling/getCanonicalFile() and re-run; if it intermittently vanishes, add a bounded retry around the whole copy

Example fix

// before
boolean ok = FileUtil.copy(srcFile, fs, dstPath, false, conf);

// after
if (!srcFile.exists() || !(srcFile.isFile() || srcFile.isDirectory())) {
  throw new FileNotFoundException(srcFile + ": not a copyable regular file/dir");
}
boolean ok = FileUtil.copy(srcFile, fs, dstPath, false, conf);
Defensive patterns

Strategy: validation

Validate before calling

private static void requireCopyableSource(File src) throws IOException {
  if (!src.exists()) throw new FileNotFoundException(src.toString());
  if (!src.isFile() && !src.isDirectory()) {
    throw new IOException(src + " is not a regular file or directory");
  }
}
// call before FileUtil.copy(src, dstFS, dst, deleteSource, conf)

Type guard

static boolean isCopyableLocalSource(File f) {
  return f.isFile() || f.isDirectory(); // excludes FIFOs, devices, sockets
}

Try / catch

try {
  FileUtil.copy(src, fs, dst, false, conf);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().endsWith("No such file or directory")) {
    // source vanished or is a special file; re-stage and retry once
  } else throw e;
}

Prevention

When it happens

Trigger: Calling FileUtil.copy(localFile, remoteFS, dstPath, deleteSource, conf) where localFile is a character device, FIFO, or socket; recursive directory copy where a listed file is unlinked between listFiles() and copy(); passing a symlink whose target is swapped mid-copy.

Common situations: Scripts copying a directory tree that another process (log rotation, tmp cleaner) is concurrently deleting; passing /dev/null or a named pipe as source; TOCTOU races in test harnesses that stage and copy files quickly.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a77e11be03a9f370. Report an issue: GitHub.