{"record":{"id":"63d4b9569a69e160","repo":"apache/hadoop","slug":"join-on-responder-thread-timed-out","errorCode":null,"errorMessage":"Join on responder thread {} timed out","messagePattern":"Join on responder thread (.+?) timed out","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java","lineNumber":1105,"sourceCode":"          responder.interrupt();\n        }\n        IOUtils.closeStream(this);\n        cleanupBlock();\n      }\n      if (responder != null) {\n        try {\n          responder.interrupt();\n          // join() on the responder should timeout a bit earlier than the\n          // configured deadline. Otherwise, the join() on this thread will\n          // likely timeout as well.\n          long joinTimeout = datanode.getDnConf().getXceiverStopTimeout();\n          joinTimeout = joinTimeout > 1  ? joinTimeout*8/10 : joinTimeout;\n          responder.join(joinTimeout);\n          if (responder.isAlive()) {\n            String msg = \"Join on responder thread \" + responder\n                + \" timed out\";\n            LOG.warn(msg + \"\\n\" + StringUtils.getStackTrace(responder));\n            throw new IOException(msg);\n          }\n        } catch (InterruptedException e) {\n          responder.interrupt();\n          // do not throw if shutting down for restart.\n          if (!datanode.isRestarting()) {\n            throw new InterruptedIOException(\"Interrupted receiveBlock\");\n          }\n        }\n        responder = null;\n      }\n    }\n  }\n\n  /**\n   * If we have downstream DNs and peerMetrics are enabled, then initialize\n   * some state for monitoring the performance of downstream DNs.\n   *\n   * @param downstreams downstream DNs, or null if there are none.","sourceCodeStart":1087,"sourceCodeEnd":1123,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L1087-L1123","documentation":"Error \"Join on responder thread {} timed out\" thrown in apache/hadoop.","triggerScenarios":"The receiver thread waits to join the PacketResponder thread after the write finished or failed, but the responder does not exit within the timeout.","commonSituations":"The PacketResponder thread did not finish within the join timeout during block write shutdown. Look for blocked downstream DataNodes or slow disks delaying acks.","solutions":["Increase dfs.datanode.handler.count or investigate why the PacketResponder thread is stuck (slow disk, blocked downstream mirror).","Check downstream DataNode logs and disk health; a hung responder usually indicates the write pipeline is stalled."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}