apache/hadoop · error · IOException

Buffer interrupted while waiting for the writer

Error message

Buffer interrupted while waiting for the writer

What it means

While a record is serialized into the sort buffer, Buffer.write blocks (under spillLock, in a do/while on blockwrite) once the soft limit (mapreduce.task.io.sort.spill.percent) is hit, waiting for the background spill to drain the buffer. If the map thread is interrupted while waiting on spillDone, this IOException is thrown - the task attempt is being killed while a spill is in progress.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java:1463

                    final int size = distanceTo(bufstart, bufindex) + len;
                    setEquator(0);
                    bufstart = bufend = bufindex = equator;
                    kvstart = kvend = kvindex;
                    bufvoid = kvbuffer.length;
                    throw new MapBufferTooSmallException(size + " bytes");
                  }
                }
              }

              if (blockwrite) {
                // wait for spill
                try {
                  while (spillInProgress) {
                    reporter.progress();
                    spillDone.await();
                  }
                } catch (InterruptedException e) {
                    throw new IOException(
                        "Buffer interrupted while waiting for the writer", e);
                }
              }
            } while (blockwrite);
          } finally {
            spillLock.unlock();
          }
        }
        // here, we know that we have sufficient space to write
        if (bufindex + len > bufvoid) {
          final int gaplen = bufvoid - bufindex;
          System.arraycopy(b, off, kvbuffer, bufindex, gaplen);
          len -= gaplen;
          off += gaplen;
          bufindex = 0;
        }
        System.arraycopy(b, off, kvbuffer, bufindex, len);
        bufindex += len;

View on GitHub (pinned to 2add963021)

Solutions

1. Find the kill source in AM/NodeManager logs; the spill-wait interruption is only the symptom
2. Rerun the job - kill races are transient
3. If spills are pathologically slow, increase mapreduce.task.io.sort.mb so spills are less frequent
4. Check disk throughput on the node if only one host is affected

When it happens

Trigger: collect() fills the buffer past the soft limit; the writer waits while spillInProgress; the thread is interrupted by a task kill (speculative loser, user/job kill, preemption) or JVM shutdown racing a large or slow spill.

Common situations: Killing a job from the CLI while maps are spilling; speculative maps killed mid-spill on slow disks; YARN preemption under capacity pressure; NM container kill with short grace periods.

Related errors

• Spill thread failed to initialize
• Interrupted while waiting for the writer
• Spill failed
• Too many spill files got created, control it with mapreduce.
• Unable to rename {src} to {dst}: couldn't create parent dire

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/636b485869d91304. Report an issue: GitHub.