apache/hadoop · error · IOException

Rec# : Failed to skip past key of length

Error message

Rec# {recNo}: Failed to skip past key of length: {currentKeyLength}

What it means

InMemoryReader iterates merged in-memory IFile segments during the reduce-side merge. After exposing the key bytes to the caller it must skip currentKeyLength bytes to position on the value; skip returning fewer bytes means the segment contains less data than the record header declared, i.e. corrupt in-memory map output. dumpOnError() dumps the segment for diagnosis before the exception is rethrown.

Solutions

Inspect the dumpOnError() output path logged with the failure to see the raw segment bytes.
Identify the map output/host involved and check that NodeManager's disk health.
Retry the job; if the same mapId fails repeatedly, force a re-run of that map (or restart the serving NM).
If reproducible, capture the dump and stack and open a MapReduce JIRA.

Defensive patterns

Strategy: try-catch

Try / catch

catch (java.io.IOException e) { if (String.valueOf(e.getMessage()).contains("Failed to skip past key")) { /* corrupt in-memory segment: dumpOnError already wrote diagnostics; retry the job */ } else { throw e; } }

Prevention

Watch reducer logs for the dumpOnError output path when merge errors appear.
Keep NM disks healthy; corrupt fetches upstream become merge-time corruption.
Report reproducible cases with the dump attached.

When it happens

Trigger: An in-memory segment built from a corrupt shuffle fetch; an InMemoryReader constructed with wrong buffer boundaries; IFile record serialization mismatch between writer and reader.

Common situations: Follows shuffle data corruption on a specific map output (disk or network); almost always framework-internal rather than direct API misuse, surfacing during the merge phase of a reduce task.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/426f67bd39fb9f63. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryReader.java:101

      fos.write(buffer, 0, bufferSize);
    } catch (IOException ioe) {
      System.err.println("Failed to dump map-output of " + taskAttemptId);
    }
  }
  
  public boolean nextRawKey(DataInputBuffer key) throws IOException {
    try {
      if (!positionToNextRecord(memDataIn)) {
        return false;
      }
      // Setup the key
      int pos = memDataIn.getPosition();
      byte[] data = memDataIn.getData();
      key.reset(data, pos, currentKeyLength);
      // Position for the next value
      long skipped = memDataIn.skip(currentKeyLength);
      if (skipped != currentKeyLength) {
        throw new IOException("Rec# " + recNo + 
            ": Failed to skip past key of length: " + 
            currentKeyLength);
      }

      // Record the byte
      bytesRead += currentKeyLength;
      return true;
    } catch (IOException ioe) {
      dumpOnError();
      throw ioe;
    }
  }
  
  public void nextRawValue(DataInputBuffer value) throws IOException {
    try {
      int pos = memDataIn.getPosition();
      byte[] data = memDataIn.getData();
      value.reset(data, pos, currentValueLength);

View on GitHub (pinned to 2add963021)