apache/hadoop · error · IOException

Possible disk error: Failed to create {}

Error message

Possible disk error: Failed to create {}

What it means

IOException (prefixed with DISK_ERROR = 'Possible disk error:') from DatanodeUtil.createFileWithExistsCheck: the exists() pre-check passed, but the actual file creation via FileIoProvider.createFile threw an IOException. Unlike 2288/2290, this is a genuine OS-level creation failure - permission denial, missing parent directory, read-only or full filesystem, or failing disk hardware - and the DataNode treats it as evidence the volume is unhealthy.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DatanodeUtil.java:74

  /**
   * Create a new file.
   * @throws IOException 
   * if the file already exists or if the file cannot be created.
   */
  public static File createFileWithExistsCheck(
      FsVolumeSpi volume, Block b, File f,
      FileIoProvider fileIoProvider) throws IOException {
    if (fileIoProvider.exists(volume, f)) {
      throw new IOException("Failed to create temporary file for " + b
          + ".  File " + f + " should not be present, but is.");
    }
    // Create the zero-length temp file
    final boolean fileCreated;
    try {
      fileCreated = fileIoProvider.createFile(volume, f);
    } catch (IOException ioe) {
      throw new IOException(DISK_ERROR + "Failed to create " + f, ioe);
    }
    if (!fileCreated) {
      throw new IOException("Failed to create temporary file for " + b
          + ".  File " + f + " should be creatable, but is already present.");
    }
    return f;
  }
  
  /**
   * @return the meta name given the block name and generation stamp.
   */
  public static String getMetaName(String blockName, long generationStamp) {
    return blockName + "_" + generationStamp + Block.METADATA_EXTENSION; 
  }

  /** @return the unlink file. */
  public static File getUnlinkTmpFile(File f) {
    return new File(f.getParentFile(), f.getName()+UNLINK_BLOCK_SUFFIX);

View on GitHub (pinned to 2add963021)

Solutions

1. Check df -h and df -i on the volume and free space, or raise dfs.datanode.du.reserved so HDFS stops before the disk fills
2. Verify permissions/ownership of the full path for the datanode user and remount the volume read-write if the kernel flipped it (dmesg for I/O errors)
3. If dmesg/SMART show hardware errors, evacuate the volume (hdfs dfsadmin -evacuate or decommission node), replace the disk, reformat the volume
4. Restart the DataNode after fixing the volume so it re-checks and re-enables the volume (see dfs.datanode.failed.volumes.tolerated)

Example fix

# before: 'Possible disk error: Failed to create /data2/dfs/current/BP-.../subdir3/blk_...' 
df -h /data2            # often 100% full
dmesg | tail            # look for I/O errors, remount-read-only

# after: free space / fix mount, then restart DN to re-enable the volume
sudo rm -rf /data2/lost+found/core.*   # reclaim space (example)
sudo mount -o remount,rw /data2
hdfs --daemon stop datanode && hdfs --daemon start datanode

// Health-gate volume writes: check space and writability before heavy write bursts
for (File vol : dataDirs) {
  long usable = vol.getUsableSpace();
  if (usable < reservedBytes /* e.g. dfs.datanode.du.reserved */) {
    throw new IOException("Volume nearly full, writes will fail: " + vol);
  }
  if (!vol.canWrite()) {
    throw new IOException("Volume not writable: " + vol);
  }
}

try {
  DatanodeUtil.createFileWithExistsCheck(volume, b, f, fileIoProvider);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Possible disk error")) {
    // surface to volume-failure handling: check df/dmesg/SMART for that volume,
    // rely on dfs.datanode.failed.volumes.tolerated, plan disk replacement
  }
}

When it happens

Trigger: fileIoProvider.createFile(volume, f) raises IOException during a new block write: ENOSPC (volume full), EACCES (ownership/permission changed under the running DataNode), EROFS (volume remounted read-only after an FS error), parent directory deleted, or I/O errors from a dying disk.

Common situations: Data volumes filled to 100% (reserved dfs.datanode.du.reserved too small); sysadmin chown/chmod on data dirs while the DN runs; disk or RAID controller failure forcing remount read-only; container deployments where the volume mount vanished.

Related errors

• Cannot list contents of {}
• Cannot remove directory {}
• Cannot remove directory {}
• Cannot create directory {}
• Mkdirs failed to create {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a197417e75544b5c. Report an issue: GitHub.