{"record":{"id":"ca1e3467dff54426","repo":"apache/hadoop","slug":"spill-thread-failed-to-initialize","errorCode":null,"errorMessage":"Spill thread failed to initialize","messagePattern":"Spill thread failed to initialize","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java","lineNumber":1078,"sourceCode":"      if (combinerRunner != null) {\n        final Counters.Counter combineOutputCounter =\n          reporter.getCounter(TaskCounter.COMBINE_OUTPUT_RECORDS);\n        combineCollector= new CombineOutputCollector<K,V>(combineOutputCounter, reporter, job);\n      } else {\n        combineCollector = null;\n      }\n      spillInProgress = false;\n      minSpillsForCombine = job.getInt(JobContext.MAP_COMBINE_MIN_SPILLS, 3);\n      spillThread.setDaemon(true);\n      spillThread.setName(\"SpillThread\");\n      spillLock.lock();\n      try {\n        spillThread.start();\n        while (!spillThreadRunning) {\n          spillDone.await();\n        }\n      } catch (InterruptedException e) {\n        throw new IOException(\"Spill thread failed to initialize\", e);\n      } finally {\n        spillLock.unlock();\n      }\n      if (sortSpillException != null) {\n        throw new IOException(\"Spill thread failed to initialize\",\n            sortSpillException);\n      }\n    }\n\n    /**\n     * Serialize the key, value to intermediate storage.\n     * When this method returns, kvindex must refer to sufficient unused\n     * storage to store one METADATA.\n     */\n    public synchronized void collect(K key, V value, final int partition\n                                     ) throws IOException {\n      reporter.progress();\n      if (key.getClass() != keyClass) {","sourceCodeStart":1060,"sourceCodeEnd":1096,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java#L1060-L1096","documentation":"MapTask's sorting map-output collector starts a background SpillThread and blocks on a lock condition (spillDone) until the thread sets spillThreadRunning. If the collector thread is interrupted during that wait, the InterruptedException is wrapped in this IOException and map-task initialization fails. It is nearly always the shadow of an external kill of the task attempt, not a defect in your mapper or data.","triggerScenarios":"spillThread.start() followed by spillDone.await() inside spillLock; the waiting thread receives interrupt(). Happens when the MR ApplicationMaster kills the attempt (speculative-execution loser, user/job kill, preemption) or the JVM is shutting down exactly during MapOutputBuffer initialization.","commonSituations":"Speculative map whose faster duplicate finishes first; job killed from CLI or by the AM during startup; NodeManager shutdown/restart or YARN preemption racing task init; test harnesses that interrupt task threads directly.","solutions":["Check AM and NodeManager logs for the kill reason (speculation, preemption, job kill) - the mapper log only shows the symptom","Rerun the job; a single occurrence is a kill-race transient","If it repeats on one host, inspect NodeManager health, restarts, and disk latency there","Reduce kill exposure: tune or disable mapreduce.map.speculative if duplicate-map kills are frequent"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Rely on MR attempt retries instead of catching this in application code - the interrupt comes from the framework killing the task","Tune or disable mapreduce.map.speculative when duplicate-map kills become common","Track KILLED vs FAILED attempt metrics separately to catch speculation/preemption pressure"],"tags":["hadoop","mapreduce","maptask","spill","threading","interrupted"],"backgroundTag":"thread-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}