{"record":{"id":"c95e5d5b7ab9cbd9","repo":"apache/hadoop","slug":"spill-failed","errorCode":null,"errorMessage":"Spill failed","messagePattern":"Spill failed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java","lineNumber":1534,"sourceCode":"          sortAndSpill();\n        }\n      } catch (InterruptedException e) {\n        throw new IOException(\"Interrupted while waiting for the writer\", e);\n      } finally {\n        spillLock.unlock();\n      }\n      assert !spillLock.isHeldByCurrentThread();\n      // shut down spill thread and wait for it to exit. Since the preceding\n      // ensures that it is finished with its work (and sortAndSpill did not\n      // throw), we elect to use an interrupt instead of setting a flag.\n      // Spilling simultaneously from this thread while the spill thread\n      // finishes its work might be both a useful way to extend this and also\n      // sufficient motivation for the latter approach.\n      try {\n        spillThread.interrupt();\n        spillThread.join();\n      } catch (InterruptedException e) {\n        throw new IOException(\"Spill failed\", e);\n      }\n      // release sort buffer before the merge\n      kvbuffer = null;\n      mergeParts();\n      Path outputPath = mapOutputFile.getOutputFile();\n      fileOutputByteCounter.increment(rfs.getFileStatus(outputPath).getLen());\n      // If necessary, make outputs permissive enough for shuffling.\n      if (!SHUFFLE_OUTPUT_PERM.equals(\n          SHUFFLE_OUTPUT_PERM.applyUMask(FsPermission.getUMask(job)))) {\n        Path indexPath = mapOutputFile.getOutputIndexFile();\n        rfs.setPermission(outputPath, SHUFFLE_OUTPUT_PERM);\n        rfs.setPermission(indexPath, SHUFFLE_OUTPUT_PERM);\n      }\n    }\n\n    public void close() { }\n\n    protected class SpillThread extends Thread {","sourceCodeStart":1516,"sourceCodeEnd":1552,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java#L1516-L1552","documentation":"flush() stops the spill thread with interrupt() and then join()s it before merging spill parts. If the collector thread is itself interrupted while joining, this IOException is thrown - a second interrupt during task teardown, i.e., the attempt is being forcibly killed even as it shuts its writer down.","triggerScenarios":"The task attempt receives another kill interrupt in the window between spillThread.interrupt() and spillThread.join() returning; aggressive teardown (job kill, NM container kill) during final merge preparation.","commonSituations":"Job killed exactly as maps finalize; NM container kill with a short grace period; repeated kills under YARN preemption.","solutions":["Identify the killer in AM/NodeManager logs around the attempt's end time","Rerun the job; double-interrupt teardown races are rare transients","If it clusters on one host, check NM container-kill behavior and grace periods there"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Configure generous container-kill grace periods if you control NM settings","Rely on job-level retries; this only fires when the task is already being killed","Investigate repeated kills on one host (NM restarts, disk health)"],"tags":["hadoop","mapreduce","spill","thread-shutdown","interrupted"],"backgroundTag":"thread-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}