apache/hadoop · error · IOException

{}seqno: expected={}, received={}

Error message

{}seqno: expected={}, received={}

What it means

Error "{}seqno: expected={}, received={}" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java:1447

                LOG.info("Relaying an out of band ack of type " + oobStatus);
                sendAckUpstream(ack, PipelineAck.UNKOWN_SEQNO, 0L, 0L,
                    PipelineAck.combineHeader(datanode.getECN(),
                      Status.SUCCESS,
                      datanode.getSLOWByBlockPoolId(block.getBlockPoolId())));
                continue;
              }
              seqno = ack.getSeqno();
            }
            if (seqno != PipelineAck.UNKOWN_SEQNO
                || type == PacketResponderType.LAST_IN_PIPELINE) {
              pkt = waitForAckHead(seqno);
              if (!isRunning()) {
                break;
              }
              expected = pkt.seqno;
              if (type == PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE
                  && seqno != expected) {
                throw new IOException(myString + "seqno: expected=" + expected
                    + ", received=" + seqno);
              }
              if (type == PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE) {
                // The total ack time includes the ack times of downstream
                // nodes.
                // The value is 0 if this responder doesn't have a downstream
                // DN in the pipeline.
                totalAckTimeNanos = ackRecvNanoTime - pkt.ackEnqueueNanoTime;
                // Report the elapsed time from ack send to ack receive minus
                // the downstream ack time.
                long ackTimeNanos = totalAckTimeNanos
                    - ack.getDownstreamAckTimeNanos();
                if (ackTimeNanos < 0) {
                  if (LOG.isDebugEnabled()) {
                    LOG.debug("Calculated invalid ack time: " + ackTimeNanos
                        + "ns.");
                  }
                } else {

View on GitHub (pinned to 2add963021)

Solutions

1. Check network ordering/loss between the sender and this DataNode; a seqno mismatch means a packet was dropped or duplicated in transit.
2. Verify client and DataNode protocol versions match; mismatched seqno tracking is a protocol error.

When it happens

Trigger: The sequence number of a received packet does not match the expected next sequence number in the write pipeline.

Common situations: Packet sequence numbers did not match on the pipeline, indicating lost or duplicated packets. Check network reliability and client retry behavior.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/151723068b0009c6. Report an issue: GitHub.