apache/hadoop · error · IOException

not able to find the highest writable parent dir

Error message

not able to find the highest writable parent dir

What it means

LocalFileSystem.reportChecksumFailure() runs after a local file fails checksum verification and tries to quarantine the corrupt file under a bad_files directory in the highest writable ancestor that stays on the same mount device (per DF). If no ancestor is both writable and on the device, this IOException is thrown while handling the original corruption.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java:124

  public boolean reportChecksumFailure(Path p, FSDataInputStream in,
                                       long inPos,
                                       FSDataInputStream sums, long sumsPos) {
    try {
      // canonicalize f
      File f = ((RawLocalFileSystem)fs).pathToFile(p).getCanonicalFile();
      
      // find highest writable parent dir of f on the same device
      String device = new DF(f, getConf()).getMount();
      File parent = f.getParentFile();
      File dir = null;
      while (parent != null && FileUtil.canWrite(parent) &&
          parent.toString().startsWith(device)) {
        dir = parent;
        parent = parent.getParentFile();
      }

      if (dir==null) {
        throw new IOException(
                              "not able to find the highest writable parent dir");
      }
        
      // move the file there
      File badDir = new File(dir, "bad_files");
      if (!badDir.mkdirs()) {
        if (!badDir.isDirectory()) {
          throw new IOException("Mkdirs failed to create " + badDir.toString());
        }
      }
      String suffix = "." + rand.nextInt();
      File badFile = new File(badDir, f.getName()+suffix);
      LOG.warn("Moving bad file " + f + " to " + badFile);
      in.close();                               // close it first
      boolean b = f.renameTo(badFile);                      // rename it
      if (!b) {
        LOG.warn("Ignoring failure of renameTo");
      }

View on GitHub (pinned to 2add963021)

Solutions

  1. Make at least one ancestor directory of the data file writable by the running user (chown/chmod)
  2. Move the data off the read-only mount onto a writable local volume
  3. Inspect `df <path>` and permission bits up the tree, fix, then retry the read (expect the file to still be corrupt — restore from a good copy)

Example fix

// before: file sits under a read-only dir; quarantine lookup fails
$ ls -ld /ro-volume/data        # root-owned, no write for daemon user
// after
$ sudo chown <procuser> /ro-volume/data && sudo chmod u+w /ro-volume/data
Defensive patterns

Strategy: try-catch

Validate before calling

File parent = pathToFile(p).getParentFile();
boolean writableAncestor = false;
while (parent != null) {
  if (FileUtil.canWrite(parent)) { writableAncestor = true; break; }
  parent = parent.getParentFile();
}
if (!writableAncestor) throw new IOException("no writable ancestor for " + p);

Try / catch

try {
  FSDataInputStream in = localFs.open(p);
} catch (IOException e) {
  // may carry 'not able to find the highest writable parent dir':
  // the underlying event is checksum corruption; restore from a clean copy
}

Prevention

When it happens

Trigger: A checksummed read on LocalFileSystem (file with a .crc sidecar) detects corruption, and every ancestor of the file up to the device root fails FileUtil.canWrite() or leaves the device prefix, so dir stays null.

Common situations: Data under root-owned or read-only directories, read-only mounts, hardened containers where the daemon user cannot write anywhere up the tree, unexpected DF mount reporting for the path.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/25631b88884a4a5e. Report an issue: GitHub.