apache/hadoop · error · IOException

Failed to copy full contents from '{srcFile}' to '{destFile}

Error message

Failed to copy full contents from '{srcFile}' to '{destFile}'

What it means

Post-copy integrity check of Storage.nativeCopyFileUnbuffered: the native copy returned without error, but srcFile.length() != destFile.length(), i.e. the destination is truncated. This catches partial writes the syscall did not report - most often because the source was still being written while copied, or the write side hit a quota/space limit silently.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java:1369

        throw new IOException("Destination '" + destFile
            + "' exists but is read-only");
      } else {
        if (destFile.delete() == false) {
          throw new IOException("Destination '" + destFile
              + "' exists but cannot be deleted");
        }
      }
    }
    try {
      NativeIO.copyFileUnbuffered(srcFile, destFile);
    } catch (NativeIOException e) {
      throw new IOException("Failed to copy " + srcFile.getCanonicalPath()
          + " to " + destFile.getCanonicalPath()
          + " due to failure in NativeIO#copyFileUnbuffered(). "
          + e.toString());
    }
    if (srcFile.length() != destFile.length()) {
      throw new IOException("Failed to copy full contents from '" + srcFile
          + "' to '" + destFile + "'");
    }
    if (preserveFileDate) {
      if (destFile.setLastModified(srcFile.lastModified()) == false) {
        LOG.debug("Failed to preserve last modified date from'{}' to '{}'",
            srcFile, destFile);
      }
    }
  }

  /**
   * Recursively delete all the content of the directory first and then 
   * the directory itself from the local filesystem.
   * @param dir The directory to delete
   * @throws IOException
   */
  public static void deleteDir(File dir) throws IOException {
    if (!FileUtil.fullyDelete(dir))

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for the writer to finish (e.g. checkpoint finalized) and confirm the source size is stable across two stats, then retry the copy
2. Check destination volume space and quotas
3. If it recurs on local disk, suspect hardware and inspect dmesg/SMART

long l1 = srcFile.length();
Thread.sleep(1000);
long l2 = srcFile.length();
if (l1 != l2) throw new IOException("source still being written: " + srcFile);

try {
  Storage.nativeCopyFileUnbuffered(src, dst, true);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Failed to copy full contents")) {
    // source was mid-write: wait for the producer to finalize, then retry the copy
  }
}

When it happens

Trigger: Copying a file that is still being appended (checkpoint not finalized); destination quota or sparse-file behavior truncating the write; NFS oddities in size reporting; source replaced mid-copy.

Common situations: Checkpoint race - copying fsimage before the NameNode finalized it; quota-limited volumes; flaky network filesystems for storage dirs.

Related errors

• Could not create DataChecksum from the byte array of length
• Could not create DataChecksum from the byte array of length
• Could not create DataChecksum of type %d with bytesPerChecks
• Trying to commit inconsistent block: id = {blockId}, expecte
• Recovery block {b} where it is not under construction.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/708f523531e9c35b. Report an issue: GitHub.