{"record":{"id":"1bb8e9f664a2b486","repo":"apache/hadoop","slug":"failed-to-shutdown-streamer","errorCode":null,"errorMessage":"Failed to shutdown streamer","messagePattern":"Failed to shutdown streamer","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java","lineNumber":849,"sourceCode":"  boolean isClosed() {\n    return closed || getStreamer().streamerClosed();\n  }\n\n  void setClosed() {\n    closed = true;\n    dfsClient.endFileLease(getUniqKey());\n    getStreamer().release();\n  }\n\n  // shutdown datastreamer and responseprocessor threads.\n  // interrupt datastreamer if force is true\n  protected void closeThreads(boolean force) throws IOException {\n    try {\n      getStreamer().close(force);\n      getStreamer().join();\n      getStreamer().closeSocket();\n    } catch (InterruptedException e) {\n      throw new IOException(\"Failed to shutdown streamer\");\n    } finally {\n      getStreamer().setSocketToNull();\n      setClosed();\n    }\n  }\n\n  /**\n   * Closes this output stream and releases any system\n   * resources associated with this stream.\n   */\n  @Override\n  public void close() throws IOException {\n    final MultipleIOException.Builder b = new MultipleIOException.Builder();\n    synchronized (this) {\n      try (TraceScope ignored = dfsClient.newPathTraceScope(\n          \"DFSOutputStream#close\", src)) {\n        closeImpl();\n      } catch (IOException e) {","sourceCodeStart":831,"sourceCodeEnd":867,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L831-L867","documentation":"closeThreads() (invoked from close()/abort() paths) stops the DataStreamer and ResponseProcessor threads: getStreamer().close(force), join(), closeSocket(). If the calling thread is interrupted during that join, the InterruptedException is swallowed and rethrown as IOException('Failed to shutdown streamer'). Note the finally block still runs (socket nulled, stream marked closed), so the stream is mostly torn down - the exception signals that shutdown did not complete cleanly and the caller's interrupt flag was cleared by the catch.","triggerScenarios":"output.close() racing an interrupt: task-cancellation frameworks calling Future.cancel(true) or Thread.interrupt() while close() is joining the streamer threads; executor shutdownNow() during a buffered close; JVM shutdown hooks or watchdog threads interrupting a thread that is inside close().","commonSituations":"Spark/Flink/MapReduce task kills interrupting the thread performing close; applications with close-timeout watchdogs that interrupt slow closes (large remaining buffer + slow DNs); thread pools torn down while background flushers are closing HDFS files.","solutions":["Clear any pending interrupt before closing: if (Thread.interrupted()) log and continue, then call close() - a clean close needs an uninterruptible window.","Restructure cancellation: set a cancel flag, let the writing thread finish its own close(), and never interrupt a thread mid-close; await its completion instead.","Do the heavy lifting before close: call hflush()/hsync() periodically so close() has little left to push and joins quickly.","If the exception already happened, retry close() once on the same stream after clearing the interrupt - the finally block marked it closed, but completeFile/lease cleanup may still need the retry; otherwise force lease recovery via 'hdfs debug recoverLease -path <file>'.","If interrupts keep arriving from a watchdog, increase its patience for close() or distinguish close-time interrupts from true cancellation."],"exampleFix":"// before\nexecutor.shutdownNow(); // interrupts a thread inside hdfsOut.close()\n\n// after - let the owning thread close without interruption\nwriter.cancelRequested = true;\nwriterThread.join(30_000); // writer loop sees the flag and closes cleanly itself\n// and inside the writer loop:\nif (cancelRequested) { hdfsOut.hflush(); hdfsOut.close(); return; }","handlingStrategy":"try-catch","validationCode":"if (Thread.interrupted()) { // clear any pending interrupt before the close window\n  LOG.warn(\"clearing pending interrupt before hdfs close\");\n}\nout.close(); // now safe from InterruptedException -> 'Failed to shutdown streamer'","typeGuard":null,"tryCatchPattern":"try {\n  out.close();\n} catch (IOException e) {\n  if (!String.valueOf(e.getMessage()).contains(\"Failed to shutdown streamer\")) throw e;\n  LOG.warn(\"interrupted during close; retrying after clearing interrupt\", e);\n  Thread.interrupted(); // clear flag set between the join and now\n  out.close(); // finally-block already marked the stream closed; this completes cleanup\n}","preventionTips":["Never Future.cancel(true)/Thread.interrupt() a thread that may be inside close().","hflush() before close so the close window is short.","Sequence shutdown: stop writers, let them close, then tear down executors/FileSystems.","If a watchdog must interrupt, give close() its own uninterruptible grace period first."],"tags":["hdfs","hdfs-client","close","thread-interrupt","streamer"],"backgroundTag":"interrupted-during-close","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}