apache/hadoop · error · IOException

Spill failed

Error message

Spill failed

What it means

flush() stops the spill thread with interrupt() and then join()s it before merging spill parts. If the collector thread is itself interrupted while joining, this IOException is thrown - a second interrupt during task teardown, i.e., the attempt is being forcibly killed even as it shuts its writer down.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java:1534

          sortAndSpill();
        }
      } catch (InterruptedException e) {
        throw new IOException("Interrupted while waiting for the writer", e);
      } finally {
        spillLock.unlock();
      }
      assert !spillLock.isHeldByCurrentThread();
      // shut down spill thread and wait for it to exit. Since the preceding
      // ensures that it is finished with its work (and sortAndSpill did not
      // throw), we elect to use an interrupt instead of setting a flag.
      // Spilling simultaneously from this thread while the spill thread
      // finishes its work might be both a useful way to extend this and also
      // sufficient motivation for the latter approach.
      try {
        spillThread.interrupt();
        spillThread.join();
      } catch (InterruptedException e) {
        throw new IOException("Spill failed", e);
      }
      // release sort buffer before the merge
      kvbuffer = null;
      mergeParts();
      Path outputPath = mapOutputFile.getOutputFile();
      fileOutputByteCounter.increment(rfs.getFileStatus(outputPath).getLen());
      // If necessary, make outputs permissive enough for shuffling.
      if (!SHUFFLE_OUTPUT_PERM.equals(
          SHUFFLE_OUTPUT_PERM.applyUMask(FsPermission.getUMask(job)))) {
        Path indexPath = mapOutputFile.getOutputIndexFile();
        rfs.setPermission(outputPath, SHUFFLE_OUTPUT_PERM);
        rfs.setPermission(indexPath, SHUFFLE_OUTPUT_PERM);
      }
    }

    public void close() { }

    protected class SpillThread extends Thread {

View on GitHub (pinned to 2add963021)

Solutions

1. Identify the killer in AM/NodeManager logs around the attempt's end time
2. Rerun the job; double-interrupt teardown races are rare transients
3. If it clusters on one host, check NM container-kill behavior and grace periods there

When it happens

Trigger: The task attempt receives another kill interrupt in the window between spillThread.interrupt() and spillThread.join() returning; aggressive teardown (job kill, NM container kill) during final merge preparation.

Common situations: Job killed exactly as maps finalize; NM container kill with a short grace period; repeated kills under YARN preemption.

Related errors

• Spill thread failed to initialize
• Buffer interrupted while waiting for the writer
• Interrupted while waiting for the writer
• Unable to rename {src} to {dst}: couldn't create parent dire
• Unable to rename {src} to {dst}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c95e5d5b7ab9cbd9. Report an issue: GitHub.