What causes "Request looked like a retry to allocate block but it already contains bytes" in apache/hadoop?

Abort the write and restart from consistent state: recover/close the lease, then append or write a new file

apache/hadoop · error · IOException

Request looked like a retry to allocate block <lastBlockInFi

Error message

Request looked like a retry to allocate block <lastBlockInFile> but it already contains <numBytes> bytes

What it means

analyzeFileState classifies addBlock retries: when the client's previous block matches the file's penultimate block, the NameNode wants to hand back the still-empty last block. If that last block already contains bytes, the 'retry' is inconsistent with NameNode state: the client is replaying an RPC from before data was written. It throws IOException rather than hand one block to two writers.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java:663

      //    the second attempts in Part I, because the first one hasn't
      //    changed the namesystem state yet.
      //    We run this analysis again in Part II where case 4 is impossible.

      BlockInfo penultimateBlock = file.getPenultimateBlock();
      if (previous == null &&
          lastBlockInFile != null &&
          lastBlockInFile.getNumBytes() >= file.getPreferredBlockSize() &&
          lastBlockInFile.isComplete()) {
        // Case 1
        if (NameNode.stateChangeLog.isDebugEnabled()) {
           NameNode.stateChangeLog.debug(
               "BLOCK* NameSystem.allocateBlock: handling block allocation" +
               " writing to a file with a complete previous block: src=" +
               src + " lastBlock=" + lastBlockInFile);
        }
      } else if (Block.matchingIdAndGenStamp(penultimateBlock, previousBlock)) {
        if (lastBlockInFile.getNumBytes() != 0) {
          throw new IOException(
              "Request looked like a retry to allocate block " +
              lastBlockInFile + " but it already contains " +
              lastBlockInFile.getNumBytes() + " bytes");
        }

        // Case 2
        // Return the last block.
        NameNode.stateChangeLog.info("BLOCK* allocateBlock: caught retry for " +
            "allocation of a new block in " + src + ". Returning previously" +
            " allocated block " + lastBlockInFile);
        long offset = file.computeFileSize();
        BlockUnderConstructionFeature uc =
            lastBlockInFile.getUnderConstructionFeature();
        onRetryBlock[0] = makeLocatedBlock(fsn, lastBlockInFile,
            uc.getExpectedStorageLocations(), offset);
        return new FileState(file, src, iip);
      } else {
        // Case 3

View on GitHub (pinned to 2add963021)

Solutions

Abort the write and restart from consistent state: recover/close the lease, then append or write a new file
Fix the client so it never re-sends addBlock after the returned block has been written to
Use DFSOutputStream semantics: never cache and replay block allocations by hand

Example fix

// before: replaying a stale allocation
LocatedBlock lb = namenode.addBlock(src, clientName, stalePrevious, fileId, null, 0); // throws

// after: re-establish the writer from authoritative state
try (FSDataOutputStream out = fs.append(path)) {
  out.write(payload);
}

Defensive patterns

Strategy: fallback

Try / catch

try {
  lb = namenode.addBlock(src, clientName, previous, fileId, null, 0);
} catch (IOException e) { // "Request looked like a retry ..."
  // client state diverged from the NameNode: abandon this writer
  writer.abort();
  try (FSDataOutputStream out = fs.append(path)) {
    /* rewrite the tail from a consistent offset */
  }
}

Prevention

Never cache and replay addBlock RPCs; use DFSOutputStream
Make application-level retries idempotent against NameNode state
On any block-allocation IOException, re-sync from the NameNode before writing more data

When it happens

Trigger: ClientProtocol.addBlock(previous=B) where B matches the penultimate block but the last block is non-empty: duplicate or replayed RPCs after the original succeeded and data flowed, or custom clients caching and re-sending block allocations.

Common situations: Application retry wrappers replaying captured RPCs; at-least-once delivery combined with manual ClientProtocol use; rare failover edge cases in old client versions.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0ae6016720f7f10f. Report an issue: GitHub.