What causes "Failed to shutdown streamer" in apache/hadoop?

Clear any pending interrupt before closing: if (Thread.interrupted()) log and continue, then call close() - a clean close needs an uninterruptible window.

apache/hadoop · error · IOException

Failed to shutdown streamer

Error message

Failed to shutdown streamer

What it means

closeThreads() (invoked from close()/abort() paths) stops the DataStreamer and ResponseProcessor threads: getStreamer().close(force), join(), closeSocket(). If the calling thread is interrupted during that join, the InterruptedException is swallowed and rethrown as IOException('Failed to shutdown streamer'). Note the finally block still runs (socket nulled, stream marked closed), so the stream is mostly torn down - the exception signals that shutdown did not complete cleanly and the caller's interrupt flag was cleared by the catch.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:849

  boolean isClosed() {
    return closed || getStreamer().streamerClosed();
  }

  void setClosed() {
    closed = true;
    dfsClient.endFileLease(getUniqKey());
    getStreamer().release();
  }

  // shutdown datastreamer and responseprocessor threads.
  // interrupt datastreamer if force is true
  protected void closeThreads(boolean force) throws IOException {
    try {
      getStreamer().close(force);
      getStreamer().join();
      getStreamer().closeSocket();
    } catch (InterruptedException e) {
      throw new IOException("Failed to shutdown streamer");
    } finally {
      getStreamer().setSocketToNull();
      setClosed();
    }
  }

  /**
   * Closes this output stream and releases any system
   * resources associated with this stream.
   */
  @Override
  public void close() throws IOException {
    final MultipleIOException.Builder b = new MultipleIOException.Builder();
    synchronized (this) {
      try (TraceScope ignored = dfsClient.newPathTraceScope(
          "DFSOutputStream#close", src)) {
        closeImpl();
      } catch (IOException e) {

View on GitHub (pinned to 2add963021)

Solutions

Clear any pending interrupt before closing: if (Thread.interrupted()) log and continue, then call close() - a clean close needs an uninterruptible window.
Restructure cancellation: set a cancel flag, let the writing thread finish its own close(), and never interrupt a thread mid-close; await its completion instead.
Do the heavy lifting before close: call hflush()/hsync() periodically so close() has little left to push and joins quickly.
If the exception already happened, retry close() once on the same stream after clearing the interrupt - the finally block marked it closed, but completeFile/lease cleanup may still need the retry; otherwise force lease recovery via 'hdfs debug recoverLease -path <file>'.
If interrupts keep arriving from a watchdog, increase its patience for close() or distinguish close-time interrupts from true cancellation.

Example fix

// before
executor.shutdownNow(); // interrupts a thread inside hdfsOut.close()

// after - let the owning thread close without interruption
writer.cancelRequested = true;
writerThread.join(30_000); // writer loop sees the flag and closes cleanly itself
// and inside the writer loop:
if (cancelRequested) { hdfsOut.hflush(); hdfsOut.close(); return; }

Defensive patterns

Strategy: try-catch

Validate before calling

if (Thread.interrupted()) { // clear any pending interrupt before the close window
  LOG.warn("clearing pending interrupt before hdfs close");
}
out.close(); // now safe from InterruptedException -> 'Failed to shutdown streamer'

Try / catch

try {
  out.close();
} catch (IOException e) {
  if (!String.valueOf(e.getMessage()).contains("Failed to shutdown streamer")) throw e;
  LOG.warn("interrupted during close; retrying after clearing interrupt", e);
  Thread.interrupted(); // clear flag set between the join and now
  out.close(); // finally-block already marked the stream closed; this completes cleanup
}

Prevention

Never Future.cancel(true)/Thread.interrupt() a thread that may be inside close().
hflush() before close so the close window is short.
Sequence shutdown: stop writers, let them close, then tear down executors/FileSystems.
If a watchdog must interrupt, give close() its own uninterruptible grace period first.

When it happens

Trigger: output.close() racing an interrupt: task-cancellation frameworks calling Future.cancel(true) or Thread.interrupt() while close() is joining the streamer threads; executor shutdownNow() during a buffered close; JVM shutdown hooks or watchdog threads interrupting a thread that is inside close().

Common situations: Spark/Flink/MapReduce task kills interrupting the thread performing close; applications with close-timeout watchdogs that interrupt slow closes (large remaining buffer + slow DNs); thread pools torn down while background flushers are closing HDFS files.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1bb8e9f664a2b486. Report an issue: GitHub.