{"record":{"id":"6f4bebb7c46300cb","repo":"apache/hadoop","slug":"interrupted-while-waiting-for-the-writer","errorCode":null,"errorMessage":"Interrupted while waiting for the writer","messagePattern":"Interrupted while waiting for the writer","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java","lineNumber":1519,"sourceCode":"        if ((kvbend + METASIZE) % kvbuffer.length !=\n            equator - (equator % METASIZE)) {\n          // spill finished\n          resetSpill();\n        }\n        if (kvindex != kvend) {\n          kvend = (kvindex + NMETA) % kvmeta.capacity();\n          bufend = bufmark;\n          LOG.info(\"Spilling map output\");\n          LOG.info(\"bufstart = \" + bufstart + \"; bufend = \" + bufmark +\n                   \"; bufvoid = \" + bufvoid);\n          LOG.info(\"kvstart = \" + kvstart + \"(\" + (kvstart * 4) +\n                   \"); kvend = \" + kvend + \"(\" + (kvend * 4) +\n                   \"); length = \" + (distanceTo(kvend, kvstart,\n                         kvmeta.capacity()) + 1) + \"/\" + maxRec);\n          sortAndSpill();\n        }\n      } catch (InterruptedException e) {\n        throw new IOException(\"Interrupted while waiting for the writer\", e);\n      } finally {\n        spillLock.unlock();\n      }\n      assert !spillLock.isHeldByCurrentThread();\n      // shut down spill thread and wait for it to exit. Since the preceding\n      // ensures that it is finished with its work (and sortAndSpill did not\n      // throw), we elect to use an interrupt instead of setting a flag.\n      // Spilling simultaneously from this thread while the spill thread\n      // finishes its work might be both a useful way to extend this and also\n      // sufficient motivation for the latter approach.\n      try {\n        spillThread.interrupt();\n        spillThread.join();\n      } catch (InterruptedException e) {\n        throw new IOException(\"Spill failed\", e);\n      }\n      // release sort buffer before the merge\n      kvbuffer = null;","sourceCodeStart":1501,"sourceCodeEnd":1537,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java#L1501-L1537","documentation":"At the end of the map phase, MapOutputBuffer.flush() performs the final spill(s) under spillLock, waiting on spillDone until the background writer catches up, then merges the spill parts. If the collector thread is interrupted during that wait, the InterruptedException is wrapped in this IOException - the attempt was being terminated at completion time.","triggerScenarios":"Task kill or JVM shutdown interrupting flush() while the final spill (kvstart != kvend drain, inline sortAndSpill) is in flight; a speculative loser killed just after emitting its last record.","commonSituations":"Job killed during map finalization; speculation killing maps that are nearly done; NM eviction racing task close(); aggressive container kill timers.","solutions":["Check AM/NodeManager logs for the kill command behind the interrupt","Rerun the job; teardown races are transient","If frequent, reduce speculation (mapreduce.map.speculative) or investigate why the AM kills late-stage attempts"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat interrupts during flush as kill symptoms; let attempt retries absorb them","Avoid killing jobs during map finalization windows when latency matters","Keep map tasks short enough that kill races at completion are rare"],"tags":["hadoop","mapreduce","spill","flush","threading","interrupted"],"backgroundTag":"thread-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}