apache/hadoop · error · IOException

Could not rename temporary file {} to {} due to failure in n

Error message

Could not rename temporary file {} to {} due to failure in native rename. {}

What it means

AtomicFileOutputStream buffers output in a sibling '<name>.tmp' file and only publishes it over the target on close(), after flush+fsync. This IOException means close() completed the data writes but publication failed: plain File.renameTo returned false, the original file was deleted (the Windows fallback path), and NativeIO.renameTo then threw a NativeIOException whose errno text is appended. All written bytes still exist in the .tmp file; the target path may now be missing.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/AtomicFileOutputStream.java:94

      triedToClose = true;
      super.close();
      success = true;
    } finally {
      if (success) {
        boolean renamed = tmpFile.renameTo(origFile);
        if (!renamed) {
          // On windows, renameTo does not replace.
          if (origFile.exists()) {
            try {
              Files.delete(origFile.toPath());
            } catch (IOException e) {
              throw new IOException("Could not delete original file " + origFile, e);
            }
          }
          try {
            NativeIO.renameTo(tmpFile, origFile);
          } catch (NativeIOException e) {
            throw new IOException("Could not rename temporary file " + tmpFile
              + " to " + origFile + " due to failure in native rename. "
              + e.toString());
          }
        }
      } else {
        if (!triedToClose) {
          // If we failed when flushing, try to close it to not leak an FD
          IOUtils.closeStream(out);
        }
        // close wasn't successful, try to delete the tmp file
        if (!tmpFile.delete()) {
          LOG.warn("Unable to delete tmp file " + tmpFile);
        }
      }
    }
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

Identify the blocker: on Linux run 'lsof <target>' and 'lsof <target>.tmp'; on Windows use Process Explorer and exclude the Hadoop data dirs from antivirus scans, then retry the operation.
Verify directory permissions for the JVM user (chmod/chown on the parent directory, the target, and any leftover .tmp file).
Ensure the target directory is a local filesystem (same filesystem as the .tmp file); move Hadoop metadata off NFS or cross-device mounts.
If close() already failed, recover manually: clear the blocker, then 'mv <target>.tmp <target>' and regenerate any checksum sidecar (e.g., MD5FileUtils.saveMD5File) instead of re-running the whole job.
For transient locks (Windows brief opens), catch the IOException and retry close() once after a short delay before surfacing it.

Example fix

// before
try (AtomicFileOutputStream out = new AtomicFileOutputStream(md5File)) {
  out.write(md5Line.getBytes(StandardCharsets.UTF_8));
} // close() may throw 'Could not rename temporary file' and leave data only in .tmp

// after - pre-flight the publish path, then recover the .tmp on failure
File tmp = new File(md5File.getParentFile(), md5File.getName() + ".tmp");
if (tmp.exists() && !tmp.delete()) throw new IOException("Stale tmp undeletable: " + tmp);
if (md5File.exists() && !md5File.canWrite()) throw new IOException("Target not replaceable: " + md5File);
try (AtomicFileOutputStream out = new AtomicFileOutputStream(md5File)) {
  out.write(md5Line.getBytes(StandardCharsets.UTF_8));
} catch (IOException e) {
  // bytes survive in tmp; republish after clearing the lock rather than losing them
  LOG.error("Publish failed; recover {} manually: {}", tmp, e.getMessage());
  throw e;
}

Defensive patterns

Strategy: validation

Validate before calling

File dst = new File(dir, "image.md5");
File tmp = new File(dst.getParentFile(), dst.getName() + ".tmp");
if (tmp.exists() && !tmp.delete()) throw new IOException("Cannot clear stale tmp: " + tmp);
if (dst.exists() && !dst.canWrite()) throw new IOException("Target not replaceable (locked?): " + dst);
if (!dst.getAbsoluteFile().getParentFile().canWrite())
  throw new IOException("Directory not writable: " + dst.getParentFile());

Try / catch

try (AtomicFileOutputStream out = new AtomicFileOutputStream(dst)) {
  out.write(data);
} catch (IOException e) {
  // message embeds tmpFile, origFile, and the NativeIOException errno text;
  // written bytes survive in <dst>.tmp - recover them after clearing the lock
  throw new IOException("Atomic publish of " + dst + " failed, recover "
      + dst + ".tmp after resolving: " + e.getMessage(), e);
}

Prevention

On Windows, exclude Hadoop data/metadata directories from antivirus, backup, and indexer scans so target files are never held open during close().
Always use try-with-resources so close() (the publish step) runs even when a write fails.
Keep atomic-published files on local filesystems, not NFS or cross-device mounts.
Delete stale '<name>.tmp' files before creating a new AtomicFileOutputStream for the same target.

When it happens

Trigger: Calling close() (directly or via try-with-resources) when the target is held open by another process (typical on Windows, where an open handle blocks replace), when the JVM user lacks write permission on the directory, when a stale '<name>.tmp' or the target is undeletable, or when the directory sits on a filesystem where rename fails (NFS, cross-device). The 'Could not delete original file' sibling error precedes it when origFile cannot be removed first.

Common situations: Writing fsimage.md5 or checkpoint metadata on Windows while antivirus/backup/indexing holds the file; read-only or root-owned dfs namenode directories after a permission change; storage on NFS mounts with non-POSIX rename semantics; leftover .tmp files from a previous crashed write.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/496559b2651481ef. Report an issue: GitHub.