{"record":{"id":"636b485869d91304","repo":"apache/hadoop","slug":"buffer-interrupted-while-waiting-for-the-writer","errorCode":null,"errorMessage":"Buffer interrupted while waiting for the writer","messagePattern":"Buffer interrupted while waiting for the writer","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java","lineNumber":1463,"sourceCode":"                    final int size = distanceTo(bufstart, bufindex) + len;\n                    setEquator(0);\n                    bufstart = bufend = bufindex = equator;\n                    kvstart = kvend = kvindex;\n                    bufvoid = kvbuffer.length;\n                    throw new MapBufferTooSmallException(size + \" bytes\");\n                  }\n                }\n              }\n\n              if (blockwrite) {\n                // wait for spill\n                try {\n                  while (spillInProgress) {\n                    reporter.progress();\n                    spillDone.await();\n                  }\n                } catch (InterruptedException e) {\n                    throw new IOException(\n                        \"Buffer interrupted while waiting for the writer\", e);\n                }\n              }\n            } while (blockwrite);\n          } finally {\n            spillLock.unlock();\n          }\n        }\n        // here, we know that we have sufficient space to write\n        if (bufindex + len > bufvoid) {\n          final int gaplen = bufvoid - bufindex;\n          System.arraycopy(b, off, kvbuffer, bufindex, gaplen);\n          len -= gaplen;\n          off += gaplen;\n          bufindex = 0;\n        }\n        System.arraycopy(b, off, kvbuffer, bufindex, len);\n        bufindex += len;","sourceCodeStart":1445,"sourceCodeEnd":1481,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java#L1445-L1481","documentation":"While a record is serialized into the sort buffer, Buffer.write blocks (under spillLock, in a do/while on blockwrite) once the soft limit (mapreduce.task.io.sort.spill.percent) is hit, waiting for the background spill to drain the buffer. If the map thread is interrupted while waiting on spillDone, this IOException is thrown - the task attempt is being killed while a spill is in progress.","triggerScenarios":"collect() fills the buffer past the soft limit; the writer waits while spillInProgress; the thread is interrupted by a task kill (speculative loser, user/job kill, preemption) or JVM shutdown racing a large or slow spill.","commonSituations":"Killing a job from the CLI while maps are spilling; speculative maps killed mid-spill on slow disks; YARN preemption under capacity pressure; NM container kill with short grace periods.","solutions":["Find the kill source in AM/NodeManager logs; the spill-wait interruption is only the symptom","Rerun the job - kill races are transient","If spills are pathologically slow, increase mapreduce.task.io.sort.mb so spills are less frequent","Check disk throughput on the node if only one host is affected"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Expect attempt kills during long spills; depend on framework retries rather than in-task exception handling","Size mapreduce.task.io.sort.mb to the mapper's output so spills are brief and rare","Alert on KILLED attempts to catch preemption/speculation pressure early"],"tags":["hadoop","mapreduce","spill","io-sort-buffer","threading","interrupted"],"backgroundTag":"thread-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}