apache/hadoop · error · ChecksumException

Checksum error reading spill index: " + indexFileName

Error message

Checksum error reading spill index: " + indexFileName

What it means

SpillRecord reads a map task's output index file (file.out.index next to the spill), which stores one 16-byte-ish record per partition followed by a trailing CRC long. After reading all partition entries through a CheckedInputStream it compares the computed checksum with the stored long; a mismatch throws ChecksumException with position -1. It means the index file on disk is corrupt or truncated.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java:84

                     String expectedIndexOwner)
      throws IOException {

    final FileSystem rfs = FileSystem.getLocal(job).getRaw();
    final FSDataInputStream in =
        SecureIOUtils.openFSDataInputStream(new File(indexFileName.toUri()
            .getRawPath()), expectedIndexOwner, null);
    try {
      final long length = rfs.getFileStatus(indexFileName).getLen();
      final int partitions = (int) length / MAP_OUTPUT_INDEX_RECORD_LENGTH;
      final int size = partitions * MAP_OUTPUT_INDEX_RECORD_LENGTH;
      buf = ByteBuffer.allocate(size);
      if (crc != null) {
        crc.reset();
        CheckedInputStream chk = new CheckedInputStream(in, crc);
        IOUtils.readFully(chk, buf.array(), 0, size);
        
        if (chk.getChecksum().getValue() != in.readLong()) {
          throw new ChecksumException("Checksum error reading spill index: " +
                                indexFileName, -1);
        }
      } else {
        IOUtils.readFully(in, buf.array(), 0, size);
      }
      entries = buf.asLongBuffer();
    } finally {
      in.close();
    }
  }

  /**
   * Return number of IndexRecord entries in this spill.
   */
  public int size() {
    return entries.capacity() / (MapTask.MAP_OUTPUT_INDEX_RECORD_LENGTH / 8);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. In most cases let the framework retry — the failed map attempt is re-executed and writes a fresh index.
2. Check free space and health of mapreduce.cluster.local.dir volumes on the node named in the error.
3. If the same attempt keeps failing on one node, blacklist/drain the node's local dirs or restart the daemon so stale attempt directories are cleaned.
4. Persistent corruption on one volume points to hardware — schedule disk replacement.

// before serving an index file: it must be readable and size-consistent
FileStatus st = fs.getFileStatus(indexPath);
if (st.getLen() == 0 || st.getLen() % MAP_OUTPUT_INDEX_RECORD_LENGTH != 0) {
  LOG.warn("Suspicious spill index " + indexPath + " len=" + st.getLen());
}

try {
  SpillRecord rec = new SpillRecord(indexFile, conf, expectedOwner);
} catch (ChecksumException e) {
  // index is corrupt: rethrow as IOException so the map attempt is retried elsewhere
  throw new IOException("Corrupt spill index " + indexFile, e);
}

When it happens

Trigger: The .index file is shorter or different than written: node killed (OOM-kill, hard shutdown) mid-spill, disk filled while writing the index, sector read errors, or leftover partial files from a crashed previous attempt being served to reducers.

Common situations: Full local disks on TaskTrackers/NodeManagers; flaky disks producing silent corruption; shuffle-serving reducers hitting an index written by an attempt that died; the error usually surfaces on the reduce side during map-output fetch.

Related errors

• Spill thread failed to initialize
• Buffer interrupted while waiting for the writer
• Interrupted while waiting for the writer
• Spill failed
• Unable to rename {src} to {dst}: couldn't create parent dire

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4e1d991d113471ce. Report an issue: GitHub.