apache/hadoop · error · IOException

Received an out-of-sequence packet for {}from {} at offset {

Error message

Received an out-of-sequence packet for {}from {} at offset {}. Expecting packet starting at {}

What it means

Error "Received an out-of-sequence packet for {}from {} at offset {}. Expecting packet starting at {}" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java:563

    return (mirrorOut == null || isDatanode || needsChecksumTranslation);
  }

  /** 
   * Receives and processes a packet. It can contain many chunks.
   * returns the number of data bytes that the packet has.
   */
  private int receivePacket() throws IOException {
    // read the next packet
    packetReceiver.receiveNextPacket(in);

    PacketHeader header = packetReceiver.getHeader();
    long seqno = header.getSeqno();
    LOG.debug("Receiving one packet for block {} seqno:{} header:{} ", block,
        seqno, header);

    // Sanity check the header
    if (header.getOffsetInBlock() > replicaInfo.getNumBytes()) {
      throw new IOException("Received an out-of-sequence packet for " + block + 
          "from " + inAddr + " at offset " + header.getOffsetInBlock() +
          ". Expecting packet starting at " + replicaInfo.getNumBytes());
    }
    if (header.getDataLen() < 0) {
      throw new IOException("Got wrong length during writeBlock(" + block + 
                            ") from " + inAddr + " at offset " + 
                            header.getOffsetInBlock() + ": " +
                            header.getDataLen()); 
    }

    long offsetInBlock = header.getOffsetInBlock();
    boolean lastPacketInBlock = header.isLastPacketInBlock();
    final int len = header.getDataLen();
    boolean syncBlock = header.getSyncBlock();

    // avoid double sync'ing on close
    if (syncBlock && lastPacketInBlock) {
      this.syncOnClose = false;

View on GitHub (pinned to 2add963021)

Solutions

1. Check network reliability between the client/upstream DataNode and this DataNode; dropped or reordered packets cause out-of-sequence offsets.
2. If caused by a flaky client or congested network, retry the write; persistent occurrences warrant investigating the TCP path and switch/NIC errors.

When it happens

Trigger: A write pipeline packet arrives whose offset does not equal the next expected offset, indicating packet loss, reordering, or a client retransmission bug.

Common situations: Packets arrived out of order on the write pipeline, usually from a retried or duplicated client send after a network error. Let the client retry; the pipeline recovers.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8448eefdc03e462a. Report an issue: GitHub.