apache/hadoop · error · IOException

Error skipping over blocks

Error message

Error skipping over blocks

What it means

With the -skipBlocks option, oiv_legacy avoids parsing each block by calling in.skipBytes(bytesToSkip) where bytesToSkip = 24 * numBlocks (three longs per block: blockId, numBytes, generation stamp - exactly the fields the non-skipping loop below the throw reads). DataInputStream.skipBytes returns fewer bytes than requested only when it hits EOF first, so this IOException means the stream ended before all declared block data could be consumed: the fsimage is truncated or internally inconsistent.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java:384

   * @param in Datastream to process
   * @param v Visitor to walk over inodes
   * @param skipBlocks Walk over each block?
   */
  private void processBlocks(DataInputStream in, ImageVisitor v,
      int numBlocks, boolean skipBlocks) throws IOException {
    v.visitEnclosingElement(ImageElement.BLOCKS,
                            ImageElement.NUM_BLOCKS, numBlocks);
    
    // directory or symlink or reference node, no blocks to process    
    if(numBlocks < 0) { 
      v.leaveEnclosingElement(); // Blocks
      return;
    }
    
    if(skipBlocks) {
      int bytesToSkip = ((Long.SIZE * 3 /* fields */) / 8 /*bits*/) * numBlocks;
      if(in.skipBytes(bytesToSkip) != bytesToSkip)
        throw new IOException("Error skipping over blocks");
      
    } else {
      for(int j = 0; j < numBlocks; j++) {
        v.visitEnclosingElement(ImageElement.BLOCK);
        v.visit(ImageElement.BLOCK_ID, in.readLong());
        v.visit(ImageElement.NUM_BYTES, in.readLong());
        v.visit(ImageElement.GENERATION_STAMP, in.readLong());
        v.leaveEnclosingElement(); // Block
      }
    }
    v.leaveEnclosingElement(); // Blocks
  }

  /**
   * Extract the INode permissions stored in the fsimage file.
   *
   * @param in Datastream to process
   * @param v Visitor to walk over inodes

View on GitHub (pinned to 2add963021)

Solutions

1. Verify integrity against the sibling checksum: md5sum -c fsimage_XXX.md5, and re-fetch the fsimage + .md5 pair from the NameNode if it fails
2. Fetch a guaranteed-complete image: hdfs dfsadmin -fs <nn> -fetchImage /tmp/fsimage_XXX after a successful checkpoint
3. Re-run without -skipBlocks to confirm the failure is positional (a full parse of a truncated file fails with a different EOF error)
4. Check transfer logs and disk space on every hop the image took

Example fix

# before
hdfs oiv_legacy -skipBlocks -p FileDistribution -i fsimage_truncated -o dist.txt
# IOException: Error skipping over blocks

# after
md5sum -c fsimage_0000000000000060000.md5   # fails -> re-fetch a complete image
hdfs dfsadmin -fs nn1.example.com:8020 -fetchImage /tmp/fsimage_0000000000000060000
hdfs oiv_legacy -skipBlocks -p FileDistribution -i /tmp/fsimage_0000000000000060000 -o dist.txt

# Verify the image is complete before any -skipBlocks run
md5sum -c fsimage_0000000000000060000.md5 || {
  echo "fsimage truncated/corrupt - re-fetch with: hdfs dfsadmin -fetchImage"; exit 1; }
hdfs oiv_legacy -skipBlocks -p FileDistribution -i fsimage_0000000000000060000 -o dist.txt

try (DataInputStream in = new DataInputStream(
        new BufferedInputStream(new FileInputStream(image)))) {
  loader.loadImage(in, visitor, /*skipBlocks*/ true);
} catch (IOException e) {
  if (e.getMessage().equals("Error skipping over blocks")) {
    // stream ended mid-image: treat as corrupt/truncated input, re-fetch the file
    throw new IOException("fsimage appears truncated - verify .md5 and re-fetch", e);
  }
  throw e;
}

When it happens

Trigger: `hdfs oiv_legacy -skipBlocks -p FileDistribution|Indented -i <fsimage>` on a file cut short mid-transfer (interrupted scp/rsync, partial NFS read, disk-full during copy), or an image whose header is corrupt so numBlocks is a garbage-huge value; in both cases skipBytes cannot return bytesToSkip.

Common situations: Copying fsimages off a NameNode while the checkpoint is still being written; archives that stored only the first N MB; images damaged in transit; occasionally a layout mismatch making the reader misinterpret a block count.

Related errors

• Negative length of the file
• Unrecognized FSImage
• Unrecognized section {s.getName()}
• Unrecognized FSImage
• Too many distribution intervals {numIntervals}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e129510b1cb82191. Report an issue: GitHub.