{"record":{"id":"426f67bd39fb9f63","repo":"apache/hadoop","slug":"rec-recno-failed-to-skip-past-key-of-length","errorCode":null,"errorMessage":"Rec# {recNo}: Failed to skip past key of length: {currentKeyLength}","messagePattern":"Rec# (.+?): Failed to skip past key of length: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryReader.java","lineNumber":101,"sourceCode":"      fos.write(buffer, 0, bufferSize);\n    } catch (IOException ioe) {\n      System.err.println(\"Failed to dump map-output of \" + taskAttemptId);\n    }\n  }\n  \n  public boolean nextRawKey(DataInputBuffer key) throws IOException {\n    try {\n      if (!positionToNextRecord(memDataIn)) {\n        return false;\n      }\n      // Setup the key\n      int pos = memDataIn.getPosition();\n      byte[] data = memDataIn.getData();\n      key.reset(data, pos, currentKeyLength);\n      // Position for the next value\n      long skipped = memDataIn.skip(currentKeyLength);\n      if (skipped != currentKeyLength) {\n        throw new IOException(\"Rec# \" + recNo + \n            \": Failed to skip past key of length: \" + \n            currentKeyLength);\n      }\n\n      // Record the byte\n      bytesRead += currentKeyLength;\n      return true;\n    } catch (IOException ioe) {\n      dumpOnError();\n      throw ioe;\n    }\n  }\n  \n  public void nextRawValue(DataInputBuffer value) throws IOException {\n    try {\n      int pos = memDataIn.getPosition();\n      byte[] data = memDataIn.getData();\n      value.reset(data, pos, currentValueLength);","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryReader.java#L83-L119","documentation":"InMemoryReader iterates merged in-memory IFile segments during the reduce-side merge. After exposing the key bytes to the caller it must skip currentKeyLength bytes to position on the value; skip returning fewer bytes means the segment contains less data than the record header declared, i.e. corrupt in-memory map output. dumpOnError() dumps the segment for diagnosis before the exception is rethrown.","triggerScenarios":"An in-memory segment built from a corrupt shuffle fetch; an InMemoryReader constructed with wrong buffer boundaries; IFile record serialization mismatch between writer and reader.","commonSituations":"Follows shuffle data corruption on a specific map output (disk or network); almost always framework-internal rather than direct API misuse, surfacing during the merge phase of a reduce task.","solutions":["Inspect the dumpOnError() output path logged with the failure to see the raw segment bytes.","Identify the map output/host involved and check that NodeManager's disk health.","Retry the job; if the same mapId fails repeatedly, force a re-run of that map (or restart the serving NM).","If reproducible, capture the dump and stack and open a MapReduce JIRA."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (java.io.IOException e) { if (String.valueOf(e.getMessage()).contains(\"Failed to skip past key\")) { /* corrupt in-memory segment: dumpOnError already wrote diagnostics; retry the job */ } else { throw e; } }","preventionTips":["Watch reducer logs for the dumpOnError output path when merge errors appear.","Keep NM disks healthy; corrupt fetches upstream become merge-time corruption.","Report reproducible cases with the dump attached."],"tags":["hadoop","mapreduce","shuffle","merge","data-corruption","ifile"],"backgroundTag":"truncated-data","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}