apache/hadoop · error · IOException

Trying to remove more than one block from file {}

Error message

Trying to remove more than one block from file {}

What it means

updateBlocks accepts at most one block removal per op, because abandonBlock removes exactly the last block. If newBlocks.length is less than oldBlocks.length minus one, replay throws IOException('Trying to remove more than one block from file <path>'). No valid client operation produces such a list, so the record is corrupt or the replay state is misaligned.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java:1182

        changeMade = true;
        fsNamesys.getBlockManager().forceCompleteBlock(oldBlock);
      }
      if (changeMade) {
        // The state or gen-stamp of the block has changed. So, we may be
        // able to process some messages from datanodes that we previously
        // were unable to process.
        fsNamesys.getBlockManager().processQueuedMessagesForBlock(newBlock);
      }
    }
    
    if (newBlocks.length < oldBlocks.length) {
      // We're removing a block from the file, e.g. abandonBlock(...)
      if (!file.isUnderConstruction()) {
        throw new IOException("Trying to remove a block from file " +
            path + " which is not under construction.");
      }
      if (newBlocks.length != oldBlocks.length - 1) {
        throw new IOException("Trying to remove more than one block from file "
            + path);
      }
      Block oldBlock = oldBlocks[oldBlocks.length - 1];
      boolean removed = FSDirWriteFileOp.unprotectedRemoveBlock(
          fsDir, path, iip, file, oldBlock);
      if (!removed && !(op instanceof UpdateBlocksOp)) {
        throw new IOException("Trying to delete non-existant block " + oldBlock);
      }
    } else if (newBlocks.length > oldBlocks.length) {
      final boolean isStriped = ecPolicy != null;
      // We're adding blocks
      for (int i = oldBlocks.length; i < newBlocks.length; i++) {
        Block newBlock = newBlocks[i];
        final BlockInfo newBI;
        if (!op.shouldCompleteLastBlock()) {
          // TODO: shouldn't this only be true for the last block?
          // what about an old-version fsync() where fsync isn't called
          // until several blocks in?

View on GitHub (pinned to 2add963021)

Solutions

1. Run 'hdfs namenode -recover' to skip the bad transaction
2. Restore a consistent fsimage plus edits from one backup
3. Trace the file's ops with 'hdfs oev' to see the sequence that produced the multi-block drop

Example fix

# before: NameNode aborts during replay
hdfs --daemon start namenode

# after: locate and skip the corrupt op
hdfs oev -i <segment> -o /tmp/edits.xml -p xml
grep -n 'SET_BLOCKS\|path=/data/x' /tmp/edits.xml | tail
hdfs namenode -recover

try {
  loader.loadFSEdits(storage, 0);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("remove more than one block")) {
    // impossible record for valid writers: treat the segment as corrupt and
    // recover or restore from backup
  }
  throw e;
}

When it happens

Trigger: A block-list update that drops two or more blocks in one op: corrupted op payload, duplicate replay shifting the lists, or logs from a faulty writer.

Common situations: Corrupt segments after crashes; hand-patched logs; replay against mismatched image state.

Related errors

• Mismatched block IDs or generation stamps for the old last b
• Mismatched block IDs or generation stamps, attempting to rep
• Trying to remove a block from file {} which is not under con
• Trying to delete non-existant block {}
• The layout version {} supports inodeId but gave bogus inodeI

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/317419c63aee263a. Report an issue: GitHub.