apache/hadoop · error · DiskOutOfSpaceException

Insufficient space for appending to ${replicaInfo}

Error message

Insufficient space for appending to ${replicaInfo}

What it means

Thrown as DiskOutOfSpaceException by FsVolumeImpl.append() (FsVolumeImpl.java:1269) when the volume's available space is less than the bytes the append still needs (estimateBlockLen minus the replica's current length). The check happens before an RBW replica with the new generation stamp is created, so the append never starts on this volume. getAvailable() already accounts for dfs.datanode.du.reserved, so 'available' is what HDFS believes is safely usable.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java:1269

      if (readBytes == -1) {
        throw new IOException("Expected to read " + checksumSize +
            " bytes from offset " + offsetInChecksum +
            " but reached end of file.");
      } else if (readBytes != checksumSize) {
        throw new IOException("Expected to read " + checksumSize +
            " bytes from offset " + offsetInChecksum + " but read " +
            readBytes + " bytes.");
      }
    }
    return lastChecksum;
  }

  public ReplicaInPipeline append(String bpid, ReplicaInfo replicaInfo,
      long newGS, long estimateBlockLen) throws IOException {

    long bytesReserved = estimateBlockLen - replicaInfo.getNumBytes();
    if (getAvailable() < bytesReserved) {
      throw new DiskOutOfSpaceException("Insufficient space for appending to "
          + replicaInfo);
    }

    assert replicaInfo.getVolume() == this:
      "The volume of the replica should be the same as this volume";

    // construct a RBW replica with the new GS
    File newBlkFile = new File(getRbwDir(bpid), replicaInfo.getBlockName());
    LocalReplicaInPipeline newReplicaInfo = new ReplicaBuilder(ReplicaState.RBW)
        .setBlockId(replicaInfo.getBlockId())
        .setLength(replicaInfo.getNumBytes())
        .setGenerationStamp(newGS)
        .setFsVolume(this)
        .setDirectoryToUse(newBlkFile.getParentFile())
        .setWriterThread(Thread.currentThread())
        .setBytesToReserve(bytesReserved)
        .buildLocalReplicaInPipeline();

View on GitHub (pinned to 2add963021)

Solutions

1. Free or add capacity on the volume(s), or lower dfs.datanode.du.reserved if it over-reserves, then let the client retry the append
2. Client side: catch DiskOutOfSpaceException during append and retry - the pipeline may pick a different volume/datanode with space
3. Run the HDFS balancer to even out volume usage so one hot volume is not the only append target
4. For very large blocks, write new data to a fresh block instead of appending to a nearly-full legacy block

Example fix

// before: append blindly; DiskOutOfSpaceException kills the writer
try (FSDataOutputStream out = fs.append(path)) { out.write(data); }

// after: pre-check the target's free space and fall back to a new block
// (client-side heuristic; the DN check remains authoritative)
long toWrite = data.length;
if (fs.getStatus(path).getRemaining() < toWrite + (64L<<20)) {
  try (FSDataOutputStream out = fs.append(path)) { out.write(data); }
} else {
  try (FSDataOutputStream out = fs.create(path, true)) { /* rewrite */ }
}

// client-side: estimate remaining bytes and compare with a coarse free-space read
long remaining = estimateBlockLen - currentBlockLen;
long free = fs.getStatus(volumePath).getRemaining();
if (free < remaining + RESERVE_MARGIN_BYTES) {
  // append will likely fail with DiskOutOfSpaceException; free space or pick another target
}

try {
  out = fs.append(path);
} catch (RemoteException re) {
  if ("java.io.DiskOutOfSpaceException".equals(re.getClassName())) {
    // free volume space / add capacity, then have the client retry the append
  } else { throw re; }
}

When it happens

Trigger: Client appends to a block whose estimated final length exceeds the receiving volume's free space; dfs.datanode.du.reserved set so high that appends of large blocks cannot reserve; many concurrent appends filling the same volume between chooser decisions.

Common situations: Small volumes combined with large max block sizes (dfs.namenode.fs-limits.max-block-files... or client-configured block sizes); reserved-for-non-DFS set aggressively to guard other processes; volumes already near full after balancer moves; appends to huge old blocks (e.g. >10GB legacy blocks).

Related errors

• Failed to create directory {}
• Xceiver count {} exceeds the limit of concurrent xceivers: {
• Mkdirs failed to create {}
• Mkdirs failed to create ${dir}
• Failed to move block file for ${this} from ${srcfile} to ${d

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/70ab7140d562d7a4. Report an issue: GitHub.