apache/hadoop · error · IOException

Could not send OOB reponse in time: {}

Error message

Could not send OOB reponse in time: {}

What it means

Error "Could not send OOB reponse in time: {}" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java:1339

     * An OOB coming from downstream will be automatically relayed upstream
     * by the responder. This method is used only by originating datanode.
     *
     * @param ackStatus the type of ack to be sent
     */
    void sendOOBResponse(final Status ackStatus) throws IOException,
        InterruptedException {
      if (!running) {
        LOG.info("Cannot send OOB response " + ackStatus + 
            ". Responder not running.");
        return;
      }

      synchronized(this) {
        if (sending) {
          wait(datanode.getOOBTimeout(ackStatus));
          // Didn't get my turn in time. Give up.
          if (sending) {
            throw new IOException("Could not send OOB reponse in time: "
                + ackStatus);
          }
        }
        sending = true;
      }

      LOG.info("Sending an out of band ack of type " + ackStatus);
      try {
        sendAckUpstreamUnprotected(null, PipelineAck.UNKOWN_SEQNO, 0L, 0L,
            PipelineAck.combineHeader(datanode.getECN(), ackStatus,
                datanode.getSLOWByBlockPoolId(block.getBlockPoolId())));
      } finally {
        // Let others send ack. Unless there are miltiple OOB send
        // calls, there can be only one waiter, the responder thread.
        // In any case, only one needs to be notified.
        synchronized(this) {
          sending = false;
          notify();

View on GitHub (pinned to 2add963021)

Solutions

1. Increase dfs.datanode.oob.timeout or investigate why the responder thread could not send the out-of-band ack promptly (disk stall, GC pause, network congestion).
2. Check for long GC pauses or slow disks on this DataNode; OOB timeouts are usually a symptom of local slowness.

When it happens

Trigger: During pipeline recovery, the DataNode fails to send an out-of-band response within the configured timeout.

Common situations: An out-of-band ack could not be sent before the deadline, usually due to an overloaded network or a slow client. Tune OOB timeouts or investigate congestion.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fce14bed8df0ff0f. Report an issue: GitHub.