apache/hadoop · error · IOException

Shutting down writer and responder since the down streams re

Error message

Shutting down writer and responder since the down streams reported the data sent by this thread is corrupt

What it means

Error "Shutting down writer and responder since the down streams reported the data sent by this thread is corrupt" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java:1665

        int h1 = PipelineAck.combineHeader(datanode.getECN(), Status.ERROR,
            datanode.getSLOWByBlockPoolId(block.getBlockPoolId()));
        replies = new int[] {h, h1};
      } else {
        short ackLen = type == PacketResponderType.LAST_IN_PIPELINE ? 0 : ack
            .getNumOfReplies();
        replies = new int[ackLen + 1];
        replies[0] = myHeader;
        for (int i = 0; i < ackLen; ++i) {
          replies[i + 1] = ack.getHeaderFlag(i);
        }
        DataNodeFaultInjector.get().markSlow(mirrorAddr, replies);
        // If the mirror has reported that it received a corrupt packet,
        // do self-destruct to mark myself bad, instead of making the
        // mirror node bad. The mirror is guaranteed to be good without
        // corrupt data on disk.
        if (ackLen > 0 && PipelineAck.getStatusFromHeader(replies[1]) ==
          Status.ERROR_CHECKSUM) {
          throw new IOException("Shutting down writer and responder "
              + "since the down streams reported the data sent by this "
              + "thread is corrupt");
        }
      }
      PipelineAck replyAck = new PipelineAck(seqno, replies,
          totalAckTimeNanos);
      if (replyAck.isSuccess()
          && offsetInBlock > replicaInfo.getBytesAcked()) {
        replicaInfo.setBytesAcked(offsetInBlock);
      }
      // send my ack back to upstream datanode
      long begin = Time.monotonicNow();
      DataNodeFaultInjector.get().delay();
      /* for test only, no-op in production system */
      DataNodeFaultInjector.get().delaySendingAckToUpstream(inAddr);
      replyAck.write(upstreamOut);
      upstreamOut.flush();
      long duration = Time.monotonicNow() - begin;

View on GitHub (pinned to 2add963021)

Solutions

1. Investigate the downstream DataNode that reported corruption: check its disks, memory, and network, and replace faulty hardware.
2. Once the faulty node is fixed or decommissioned, the NameNode will re-replicate the block; no manual repair of this node is needed.

When it happens

Trigger: A downstream DataNode in the write pipeline reported that data mirrored from this node is corrupt, so this node shuts down its writer and responder threads.

Common situations: Downstream DataNodes reported corrupt data for this block, so the writer shut itself down. Investigate the downstream node logs for the original checksum failure.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ea1009cb06436a5e. Report an issue: GitHub.