apache/hadoop · error · IOException

Spill thread failed to initialize

Error message

Spill thread failed to initialize

What it means

MapTask's sorting map-output collector starts a background SpillThread and blocks on a lock condition (spillDone) until the thread sets spillThreadRunning. If the collector thread is interrupted during that wait, the InterruptedException is wrapped in this IOException and map-task initialization fails. It is nearly always the shadow of an external kill of the task attempt, not a defect in your mapper or data.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java:1078

      if (combinerRunner != null) {
        final Counters.Counter combineOutputCounter =
          reporter.getCounter(TaskCounter.COMBINE_OUTPUT_RECORDS);
        combineCollector= new CombineOutputCollector<K,V>(combineOutputCounter, reporter, job);
      } else {
        combineCollector = null;
      }
      spillInProgress = false;
      minSpillsForCombine = job.getInt(JobContext.MAP_COMBINE_MIN_SPILLS, 3);
      spillThread.setDaemon(true);
      spillThread.setName("SpillThread");
      spillLock.lock();
      try {
        spillThread.start();
        while (!spillThreadRunning) {
          spillDone.await();
        }
      } catch (InterruptedException e) {
        throw new IOException("Spill thread failed to initialize", e);
      } finally {
        spillLock.unlock();
      }
      if (sortSpillException != null) {
        throw new IOException("Spill thread failed to initialize",
            sortSpillException);
      }
    }

    /**
     * Serialize the key, value to intermediate storage.
     * When this method returns, kvindex must refer to sufficient unused
     * storage to store one METADATA.
     */
    public synchronized void collect(K key, V value, final int partition
                                     ) throws IOException {
      reporter.progress();
      if (key.getClass() != keyClass) {

View on GitHub (pinned to 2add963021)

Solutions

1. Check AM and NodeManager logs for the kill reason (speculation, preemption, job kill) - the mapper log only shows the symptom
2. Rerun the job; a single occurrence is a kill-race transient
3. If it repeats on one host, inspect NodeManager health, restarts, and disk latency there
4. Reduce kill exposure: tune or disable mapreduce.map.speculative if duplicate-map kills are frequent

When it happens

Trigger: spillThread.start() followed by spillDone.await() inside spillLock; the waiting thread receives interrupt(). Happens when the MR ApplicationMaster kills the attempt (speculative-execution loser, user/job kill, preemption) or the JVM is shutting down exactly during MapOutputBuffer initialization.

Common situations: Speculative map whose faster duplicate finishes first; job killed from CLI or by the AM during startup; NodeManager shutdown/restart or YARN preemption racing task init; test harnesses that interrupt task threads directly.

Related errors

• Buffer interrupted while waiting for the writer
• Interrupted while waiting for the writer
• Spill failed
• Unable to rename {src} to {dst}: couldn't create parent dire
• Unable to rename {src} to {dst}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ca1e3467dff54426. Report an issue: GitHub.