{"record":{"id":"a197417e75544b5c","repo":"apache/hadoop","slug":"possible-disk-error-failed-to-create","errorCode":null,"errorMessage":"Possible disk error: Failed to create {}","messagePattern":"Possible disk error: Failed to create (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DatanodeUtil.java","lineNumber":74,"sourceCode":"\n  /**\n   * Create a new file.\n   * @throws IOException \n   * if the file already exists or if the file cannot be created.\n   */\n  public static File createFileWithExistsCheck(\n      FsVolumeSpi volume, Block b, File f,\n      FileIoProvider fileIoProvider) throws IOException {\n    if (fileIoProvider.exists(volume, f)) {\n      throw new IOException(\"Failed to create temporary file for \" + b\n          + \".  File \" + f + \" should not be present, but is.\");\n    }\n    // Create the zero-length temp file\n    final boolean fileCreated;\n    try {\n      fileCreated = fileIoProvider.createFile(volume, f);\n    } catch (IOException ioe) {\n      throw new IOException(DISK_ERROR + \"Failed to create \" + f, ioe);\n    }\n    if (!fileCreated) {\n      throw new IOException(\"Failed to create temporary file for \" + b\n          + \".  File \" + f + \" should be creatable, but is already present.\");\n    }\n    return f;\n  }\n  \n  /**\n   * @return the meta name given the block name and generation stamp.\n   */\n  public static String getMetaName(String blockName, long generationStamp) {\n    return blockName + \"_\" + generationStamp + Block.METADATA_EXTENSION; \n  }\n\n  /** @return the unlink file. */\n  public static File getUnlinkTmpFile(File f) {\n    return new File(f.getParentFile(), f.getName()+UNLINK_BLOCK_SUFFIX);","sourceCodeStart":56,"sourceCodeEnd":92,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DatanodeUtil.java#L56-L92","documentation":"IOException (prefixed with DISK_ERROR = 'Possible disk error:') from DatanodeUtil.createFileWithExistsCheck: the exists() pre-check passed, but the actual file creation via FileIoProvider.createFile threw an IOException. Unlike 2288/2290, this is a genuine OS-level creation failure - permission denial, missing parent directory, read-only or full filesystem, or failing disk hardware - and the DataNode treats it as evidence the volume is unhealthy.","triggerScenarios":"fileIoProvider.createFile(volume, f) raises IOException during a new block write: ENOSPC (volume full), EACCES (ownership/permission changed under the running DataNode), EROFS (volume remounted read-only after an FS error), parent directory deleted, or I/O errors from a dying disk.","commonSituations":"Data volumes filled to 100% (reserved dfs.datanode.du.reserved too small); sysadmin chown/chmod on data dirs while the DN runs; disk or RAID controller failure forcing remount read-only; container deployments where the volume mount vanished.","solutions":["Check df -h and df -i on the volume and free space, or raise dfs.datanode.du.reserved so HDFS stops before the disk fills","Verify permissions/ownership of the full path for the datanode user and remount the volume read-write if the kernel flipped it (dmesg for I/O errors)","If dmesg/SMART show hardware errors, evacuate the volume (hdfs dfsadmin -evacuate or decommission node), replace the disk, reformat the volume","Restart the DataNode after fixing the volume so it re-checks and re-enables the volume (see dfs.datanode.failed.volumes.tolerated)"],"exampleFix":"# before: 'Possible disk error: Failed to create /data2/dfs/current/BP-.../subdir3/blk_...' \ndf -h /data2            # often 100% full\ndmesg | tail            # look for I/O errors, remount-read-only\n\n# after: free space / fix mount, then restart DN to re-enable the volume\nsudo rm -rf /data2/lost+found/core.*   # reclaim space (example)\nsudo mount -o remount,rw /data2\nhdfs --daemon stop datanode && hdfs --daemon start datanode","handlingStrategy":"try-catch","validationCode":"// Health-gate volume writes: check space and writability before heavy write bursts\nfor (File vol : dataDirs) {\n  long usable = vol.getUsableSpace();\n  if (usable < reservedBytes /* e.g. dfs.datanode.du.reserved */) {\n    throw new IOException(\"Volume nearly full, writes will fail: \" + vol);\n  }\n  if (!vol.canWrite()) {\n    throw new IOException(\"Volume not writable: \" + vol);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  DatanodeUtil.createFileWithExistsCheck(volume, b, f, fileIoProvider);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Possible disk error\")) {\n    // surface to volume-failure handling: check df/dmesg/SMART for that volume,\n    // rely on dfs.datanode.failed.volumes.tolerated, plan disk replacement\n  }\n}","preventionTips":["Set dfs.datanode.du.reserved so volumes never reach 100% and ENOSPC mid-write","Alert on volume usable space and on DataNode LastVolumeFailure metrics","Lock down ownership of storage dirs so nothing re-chmods them under the running DN","Watch dmesg/SMART for read-only remounts and I/O errors; replace failing disks before HDFS notices for you"],"tags":["hdfs","datanode","disk-error","enospc","permissions","volume-health"],"backgroundTag":"disk-io-error","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}