apache/hadoop · error · IOException

Update {} (size={}) to a smaller size block {} (size={})

Error message

Update {} (size={}) to a smaller size block {} (size={})

What it means

updateBlock rejects shrinking a block: if newBlock.getNumBytes() is less than the last block's current length, the update is refused. Data already reported cannot be un-reported via updateBlock; length reduction is only legal through the truncate API.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:6208

    final INodeFile pendingFile = checkUCBlock(oldBlock, clientName);
    final String src = pendingFile.getFullPathName();
    final BlockInfo lastBlock = pendingFile.getLastBlock();
    assert !lastBlock.isComplete();

    // check new GS & length: this is not expected
    if (newBlock.getGenerationStamp() <= lastBlock.getGenerationStamp()) {
      final String msg = "Update " + oldBlock + " but the new block " + newBlock
          + " does not have a larger generation stamp than the last block "
          + lastBlock;
      LOG.warn(msg);
      throw new IOException(msg);
    }
    if (newBlock.getNumBytes() < lastBlock.getNumBytes()) {
      final String msg = "Update " + oldBlock + " (size="
          + oldBlock.getNumBytes() + ") to a smaller size block " + newBlock
          + " (size=" + newBlock.getNumBytes() + ")";
      LOG.warn(msg);
      throw new IOException(msg);
    }

    // Update old block with the new generation stamp and new length
    blockManager.updateLastBlock(lastBlock, newBlock);

    // find the DatanodeDescriptor objects
    final DatanodeStorageInfo[] storages = blockManager.getDatanodeManager()
        .getDatanodeStorageInfos(newNodes, newStorageIDs,
            "src=%s, oldBlock=%s, newBlock=%s, clientName=%s",
            src, oldBlock, newBlock, clientName);
    lastBlock.getUnderConstructionFeature().setExpectedLocations(lastBlock,
        storages, lastBlock.getBlockType());

    FSDirWriteFileOp.persistBlocks(dir, src, pendingFile, logRetryCache);
  }

  /**
   * Register a Backup name-node, verifying that it belongs

View on GitHub (pinned to 2add963021)

Solutions

1. Use the truncate API (dfs.truncate / ClientProtocol.truncate) to shorten a file
2. In recovery paths, never send a new length below the current one; re-read the block's length first
3. Validate new GS and length against the LocatedBlock before submitting

Example fix

// before
ExtendedBlock nb = new ExtendedBlock(bp, blockId, smallerLen, newGs);
nn.updateBlock(pendingFile, lastBlock, nb, false);

// after
if (smallerLen < lastBlock.getNumBytes()) {
  dfs.truncate(path, smallerLen); // proper way to shrink
} else {
  ExtendedBlock nb = new ExtendedBlock(bp, blockId, newLen, newGs);
  nn.updateBlock(pendingFile, lastBlock, nb, false);
}

if (newBlock.getNumBytes() < lastBlock.getNumBytes()) {
  throw new IllegalArgumentException(
      "updateBlock cannot shrink a block; use dfs.truncate instead");
}

try {
  updateBlock(oldBlock, newBlock, ...);
} catch (IOException e) {
  if (e.getMessage().contains("smaller size block")) {
    dfs.truncate(path, targetLen); // proper shrink path
  } else { throw e; }
}

When it happens

Trigger: Pipeline recovery retry that carries a smaller length after a partial failure; client code attempting to truncate by submitting an updateBlock with fewer bytes.

Common situations: Hand-rolled recovery logic confusing truncate with block update; retries racing replica length reports.

Related errors

• "Cannot find a block policy with the name " + policyName
• Failed to TRUNCATE_FILE <src> for <clientName> on <clientMac
• Can't have more than {} in an AddCloseOp.
• Path part {s} from URI {p} is not a valid filename.
• Not implemented by the {} FileSystem implementation

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4d5b570162001abb. Report an issue: GitHub.