{"record":{"id":"a3d33988ff5cfa8e","repo":"apache/hadoop","slug":"the-last-block-for-file-is-full","errorCode":null,"errorMessage":"The last block for file {} is full.","messagePattern":"The last block for file (.+?) is full\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java","lineNumber":384,"sourceCode":"          progress, checksum, cachingStrategy, byteArrayManager, favoredNodes,\n          addBlockFlags);\n    }\n  }\n\n  private void adjustPacketChunkSize(LocatedBlock lastBlock) throws IOException{\n\n    long usedInLastBlock = lastBlock.getBlockSize();\n    int freeInLastBlock = (int)(blockSize - usedInLastBlock);\n\n    // calculate the amount of free space in the pre-existing\n    // last crc chunk\n    int usedInCksum = (int)(lastBlock.getBlockSize() % bytesPerChecksum);\n    int freeInCksum = bytesPerChecksum - usedInCksum;\n\n    // if there is space in the last block, then we have to\n    // append to that block\n    if (freeInLastBlock == blockSize) {\n      throw new IOException(\"The last block for file \" +\n          src + \" is full.\");\n    }\n\n    if (usedInCksum > 0 && freeInCksum > 0) {\n      // if there is space in the last partial chunk, then\n      // setup in such a way that the next packet will have only\n      // one chunk that fills up the partial chunk.\n      //\n      computePacketChunkSize(0, freeInCksum);\n      setChecksumBufSize(freeInCksum);\n      getStreamer().setAppendChunk(true);\n    } else {\n      // if the remaining space in the block is smaller than\n      // that expected size of of a packet, then create\n      // smaller size packet.\n      //\n      computePacketChunkSize(\n          Math.min(dfsClient.getConf().getWritePacketSize(), freeInLastBlock),","sourceCodeStart":366,"sourceCodeEnd":402,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L366-L402","documentation":"Thrown by adjustPacketChunkSize() on the append path (append without NEW_BLOCK, last block present): it computes freeInLastBlock = blockSize - lastBlock.getBlockSize() and throws IOException('The last block for file X is full.') when freeInLastBlock == blockSize - which by the arithmetic means lastBlock.getBlockSize() == 0, i.e. the last block is EMPTY, not full (the message is misleading). The client cannot set up the append checksum state for a zero-byte block and refuses to append into it. A normal full last block is not the trigger; that case is handled by allocating a new block.","triggerScenarios":"fs.append(path) (without CreateFlag.NEW_BLOCK) on a file whose last block is 0 bytes - typically a file left behind by a crashed writer whose block was allocated but never got acked data, then finalized empty by lease recovery; also files produced by older buggy writers or partial-copy tooling.","commonSituations":"After kill -9 of a writer plus lease recovery (or 'hdfs debug recoverLease'), the file ends with an empty finalized block and the next append fails with this exact message; append jobs on directories of writer-crash artifacts failing one-by-one; restored/distcp'd files that preserved an empty tail block.","solutions":["Append with a new block instead of resuming the empty one: ((DistributedFileSystem) fs).append(path, bufferSize, EnumSet.of(CreateFlag.NEW_BLOCK)) - the empty last block is then treated as complete and data goes to a fresh block.","Confirm the diagnosis with 'hdfs fsck <path> -files -blocks' - you should see a 0-byte last block.","If NEW_BLOCK is not acceptable (tools that call plain fs.append), repair the file: read its bytes and rewrite it (distcp or copy to temp + rename), then append.","Prevent recurrence: ensure writers close cleanly and rely on lease recovery plus this check rather than attempting to append into empty blocks."],"exampleFix":"// before\nFSDataOutputStream out = fs.append(path); // IOException: last block ... is full (empty)\n\n// after\nFSDataOutputStream out = ((DistributedFileSystem) fs)\n    .append(path, 4096, EnumSet.of(CreateFlag.NEW_BLOCK));","handlingStrategy":"fallback","validationCode":"List<LocatedBlock> blocks = ((DistributedFileSystem) fs).getClient()\n    .getLocatedBlocks(path, 0).getLocatedBlocks();\nboolean emptyLastBlock = !blocks.isEmpty()\n    && blocks.get(blocks.size() - 1).getBlockSize() == 0;\nEnumSet<CreateFlag> flags = emptyLastBlock\n    ? EnumSet.of(CreateFlag.APPEND, CreateFlag.NEW_BLOCK) // fresh block, skip the empty one\n    : EnumSet.of(CreateFlag.APPEND);\nFSDataOutputStream out = ((DistributedFileSystem) fs).append(path, 4096, flags);","typeGuard":null,"tryCatchPattern":"try {\n  out = fs.append(path);\n} catch (IOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"is full\")) {\n    out = ((DistributedFileSystem) fs).append(path, 4096,\n        EnumSet.of(CreateFlag.NEW_BLOCK)); // fall back to a new block\n  } else throw e;\n}","preventionTips":["After a writer crash plus lease recovery, inspect blocks ('hdfs fsck -files -blocks') before appending.","Use CreateFlag.NEW_BLOCK appends for files with suspect tail blocks.","Ensure writers close cleanly so empty tail blocks are not left behind.","Remember the message text is misleading: 'full' here actually means a 0-byte last block."],"tags":["hdfs","hdfs-client","append","block","lease-recovery"],"backgroundTag":"append-empty-last-block","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}