apache/hadoop · error · IOException

Rec# {recNo}: Failed to skip past key of length: {currentKey

Error message

Rec# {recNo}: Failed to skip past key of length: {currentKeyLength}

What it means

InMemoryReader iterates merged in-memory IFile segments during the reduce-side merge. After exposing the key bytes to the caller it must skip currentKeyLength bytes to position on the value; skip returning fewer bytes means the segment contains less data than the record header declared, i.e. corrupt in-memory map output. dumpOnError() dumps the segment for diagnosis before the exception is rethrown.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryReader.java:101

      fos.write(buffer, 0, bufferSize);
    } catch (IOException ioe) {
      System.err.println("Failed to dump map-output of " + taskAttemptId);
    }
  }
  
  public boolean nextRawKey(DataInputBuffer key) throws IOException {
    try {
      if (!positionToNextRecord(memDataIn)) {
        return false;
      }
      // Setup the key
      int pos = memDataIn.getPosition();
      byte[] data = memDataIn.getData();
      key.reset(data, pos, currentKeyLength);
      // Position for the next value
      long skipped = memDataIn.skip(currentKeyLength);
      if (skipped != currentKeyLength) {
        throw new IOException("Rec# " + recNo + 
            ": Failed to skip past key of length: " + 
            currentKeyLength);
      }

      // Record the byte
      bytesRead += currentKeyLength;
      return true;
    } catch (IOException ioe) {
      dumpOnError();
      throw ioe;
    }
  }
  
  public void nextRawValue(DataInputBuffer value) throws IOException {
    try {
      int pos = memDataIn.getPosition();
      byte[] data = memDataIn.getData();
      value.reset(data, pos, currentValueLength);

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the dumpOnError() output path logged with the failure to see the raw segment bytes.
2. Identify the map output/host involved and check that NodeManager's disk health.
3. Retry the job; if the same mapId fails repeatedly, force a re-run of that map (or restart the serving NM).
4. If reproducible, capture the dump and stack and open a MapReduce JIRA.

catch (java.io.IOException e) { if (String.valueOf(e.getMessage()).contains("Failed to skip past key")) { /* corrupt in-memory segment: dumpOnError already wrote diagnostics; retry the job */ } else { throw e; } }

When it happens

Trigger: An in-memory segment built from a corrupt shuffle fetch; an InMemoryReader constructed with wrong buffer boundaries; IFile record serialization mismatch between writer and reader.

Common situations: Follows shuffle data corruption on a specific map output (disk or network); almost always framework-internal rather than direct API misuse, surfacing during the merge phase of a reduce task.

Related errors

• Rec# {recNo}: Failed to skip past value of length: {currentV
• Unexpected extra bytes from input stream for {mapId}
• Negative key-length not allowed: {keyLength} for {key}
• Negative value-length not allowed: {valueLength} for {value}
• Invalid configuration: maxSingleShuffleLimit should be less

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/426f67bd39fb9f63. Report an issue: GitHub.