{"record":{"id":"4e1d991d113471ce","repo":"apache/hadoop","slug":"checksum-error-reading-spill-index-indexfilen","errorCode":null,"errorMessage":"Checksum error reading spill index: \" + indexFileName","messagePattern":"Checksum error reading spill index: \" \\+ indexFileName","errorType":"exception","errorClass":"ChecksumException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java","lineNumber":84,"sourceCode":"                     String expectedIndexOwner)\n      throws IOException {\n\n    final FileSystem rfs = FileSystem.getLocal(job).getRaw();\n    final FSDataInputStream in =\n        SecureIOUtils.openFSDataInputStream(new File(indexFileName.toUri()\n            .getRawPath()), expectedIndexOwner, null);\n    try {\n      final long length = rfs.getFileStatus(indexFileName).getLen();\n      final int partitions = (int) length / MAP_OUTPUT_INDEX_RECORD_LENGTH;\n      final int size = partitions * MAP_OUTPUT_INDEX_RECORD_LENGTH;\n      buf = ByteBuffer.allocate(size);\n      if (crc != null) {\n        crc.reset();\n        CheckedInputStream chk = new CheckedInputStream(in, crc);\n        IOUtils.readFully(chk, buf.array(), 0, size);\n        \n        if (chk.getChecksum().getValue() != in.readLong()) {\n          throw new ChecksumException(\"Checksum error reading spill index: \" +\n                                indexFileName, -1);\n        }\n      } else {\n        IOUtils.readFully(in, buf.array(), 0, size);\n      }\n      entries = buf.asLongBuffer();\n    } finally {\n      in.close();\n    }\n  }\n\n  /**\n   * Return number of IndexRecord entries in this spill.\n   */\n  public int size() {\n    return entries.capacity() / (MapTask.MAP_OUTPUT_INDEX_RECORD_LENGTH / 8);\n  }\n","sourceCodeStart":66,"sourceCodeEnd":102,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java#L66-L102","documentation":"SpillRecord reads a map task's output index file (file.out.index next to the spill), which stores one 16-byte-ish record per partition followed by a trailing CRC long. After reading all partition entries through a CheckedInputStream it compares the computed checksum with the stored long; a mismatch throws ChecksumException with position -1. It means the index file on disk is corrupt or truncated.","triggerScenarios":"The .index file is shorter or different than written: node killed (OOM-kill, hard shutdown) mid-spill, disk filled while writing the index, sector read errors, or leftover partial files from a crashed previous attempt being served to reducers.","commonSituations":"Full local disks on TaskTrackers/NodeManagers; flaky disks producing silent corruption; shuffle-serving reducers hitting an index written by an attempt that died; the error usually surfaces on the reduce side during map-output fetch.","solutions":["In most cases let the framework retry — the failed map attempt is re-executed and writes a fresh index.","Check free space and health of mapreduce.cluster.local.dir volumes on the node named in the error.","If the same attempt keeps failing on one node, blacklist/drain the node's local dirs or restart the daemon so stale attempt directories are cleaned.","Persistent corruption on one volume points to hardware — schedule disk replacement."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before serving an index file: it must be readable and size-consistent\nFileStatus st = fs.getFileStatus(indexPath);\nif (st.getLen() == 0 || st.getLen() % MAP_OUTPUT_INDEX_RECORD_LENGTH != 0) {\n  LOG.warn(\"Suspicious spill index \" + indexPath + \" len=\" + st.getLen());\n}","typeGuard":null,"tryCatchPattern":"try {\n  SpillRecord rec = new SpillRecord(indexFile, conf, expectedOwner);\n} catch (ChecksumException e) {\n  // index is corrupt: rethrow as IOException so the map attempt is retried elsewhere\n  throw new IOException(\"Corrupt spill index \" + indexFile, e);\n}","preventionTips":["Monitor free space under mapreduce.cluster.local.dir; spills fail when disks fill.","Let task retries handle transient corruption instead of hand-editing index files.","Replace or drain nodes that repeatedly produce checksum errors."],"tags":["hadoop","mapreduce","shuffle","spill","checksum","disk-corruption"],"backgroundTag":"checksum-corruption","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}