What causes "The last block for file {} is full." in apache/hadoop?

Append with a new block instead of resuming the empty one: ((DistributedFileSystem) fs).append(path, bufferSize, EnumSet.of(CreateFlag.NEW_BLOCK)) - the empty last block is then treated as complete and data goes to a fresh block.

apache/hadoop · error · IOException

The last block for file {} is full.

Error message

The last block for file {} is full.

What it means

Thrown by adjustPacketChunkSize() on the append path (append without NEW_BLOCK, last block present): it computes freeInLastBlock = blockSize - lastBlock.getBlockSize() and throws IOException('The last block for file X is full.') when freeInLastBlock == blockSize - which by the arithmetic means lastBlock.getBlockSize() == 0, i.e. the last block is EMPTY, not full (the message is misleading). The client cannot set up the append checksum state for a zero-byte block and refuses to append into it. A normal full last block is not the trigger; that case is handled by allocating a new block.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:384

          progress, checksum, cachingStrategy, byteArrayManager, favoredNodes,
          addBlockFlags);
    }
  }

  private void adjustPacketChunkSize(LocatedBlock lastBlock) throws IOException{

    long usedInLastBlock = lastBlock.getBlockSize();
    int freeInLastBlock = (int)(blockSize - usedInLastBlock);

    // calculate the amount of free space in the pre-existing
    // last crc chunk
    int usedInCksum = (int)(lastBlock.getBlockSize() % bytesPerChecksum);
    int freeInCksum = bytesPerChecksum - usedInCksum;

    // if there is space in the last block, then we have to
    // append to that block
    if (freeInLastBlock == blockSize) {
      throw new IOException("The last block for file " +
          src + " is full.");
    }

    if (usedInCksum > 0 && freeInCksum > 0) {
      // if there is space in the last partial chunk, then
      // setup in such a way that the next packet will have only
      // one chunk that fills up the partial chunk.
      //
      computePacketChunkSize(0, freeInCksum);
      setChecksumBufSize(freeInCksum);
      getStreamer().setAppendChunk(true);
    } else {
      // if the remaining space in the block is smaller than
      // that expected size of of a packet, then create
      // smaller size packet.
      //
      computePacketChunkSize(
          Math.min(dfsClient.getConf().getWritePacketSize(), freeInLastBlock),

View on GitHub (pinned to 2add963021)

Solutions

Append with a new block instead of resuming the empty one: ((DistributedFileSystem) fs).append(path, bufferSize, EnumSet.of(CreateFlag.NEW_BLOCK)) - the empty last block is then treated as complete and data goes to a fresh block.
Confirm the diagnosis with 'hdfs fsck <path> -files -blocks' - you should see a 0-byte last block.
If NEW_BLOCK is not acceptable (tools that call plain fs.append), repair the file: read its bytes and rewrite it (distcp or copy to temp + rename), then append.
Prevent recurrence: ensure writers close cleanly and rely on lease recovery plus this check rather than attempting to append into empty blocks.

Example fix

// before
FSDataOutputStream out = fs.append(path); // IOException: last block ... is full (empty)

// after
FSDataOutputStream out = ((DistributedFileSystem) fs)
    .append(path, 4096, EnumSet.of(CreateFlag.NEW_BLOCK));

Defensive patterns

Strategy: fallback

Validate before calling

List<LocatedBlock> blocks = ((DistributedFileSystem) fs).getClient()
    .getLocatedBlocks(path, 0).getLocatedBlocks();
boolean emptyLastBlock = !blocks.isEmpty()
    && blocks.get(blocks.size() - 1).getBlockSize() == 0;
EnumSet<CreateFlag> flags = emptyLastBlock
    ? EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK) // fresh block, skip the empty one
    : EnumSet.of(CreateFlag.APPEND);
FSDataOutputStream out = ((DistributedFileSystem) fs).append(path, 4096, flags);

Try / catch

try {
  out = fs.append(path);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("is full")) {
    out = ((DistributedFileSystem) fs).append(path, 4096,
        EnumSet.of(CreateFlag.NEW_BLOCK)); // fall back to a new block
  } else throw e;
}

Prevention

After a writer crash plus lease recovery, inspect blocks ('hdfs fsck -files -blocks') before appending.
Use CreateFlag.NEW_BLOCK appends for files with suspect tail blocks.
Ensure writers close cleanly so empty tail blocks are not left behind.
Remember the message text is misleading: 'full' here actually means a 0-byte last block.

When it happens

Trigger: fs.append(path) (without CreateFlag.NEW_BLOCK) on a file whose last block is 0 bytes - typically a file left behind by a crashed writer whose block was allocated but never got acked data, then finalized empty by lease recovery; also files produced by older buggy writers or partial-copy tooling.

Common situations: After kill -9 of a writer plus lease recovery (or 'hdfs debug recoverLease'), the file ends with an empty finalized block and the next append fails with this exact message; append jobs on directories of writer-crash artifacts failing one-by-one; restored/distcp'd files that preserved an empty tail block.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a3d33988ff5cfa8e. Report an issue: GitHub.