apache/hadoop · warning · IOException

Interrupted while serving request. Aborting.

Error message

Interrupted while serving request. Aborting.

What it means

IOException thrown while a DataXceiver thread waits for its block pool to register with the NameNode: the wait loop sleeps one second between registration attempts, and if the thread is interrupted during that sleep it aborts the request with this message. The interruption itself is the signal - normally it means the DataNode is shutting down (its threads are interrupted) while a client request was still waiting for the block pool to become ready.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java:1429

      // not registered
    }

    // retry
    long bpReadyTimeout = dnConf.getBpReadyTimeout();
    StopWatch sw = new StopWatch();
    sw.start();
    while (sw.now(TimeUnit.SECONDS) <= bpReadyTimeout) {
      try {
        datanode.getDNRegistrationForBP(bpId);
        return;
      } catch (IOException ioe) {
        // not registered
      }
      // sleep before trying again
      try {
        Thread.sleep(1000);
      } catch (InterruptedException ie) {
        throw new IOException("Interrupted while serving request. Aborting.");
      }
    }
    // failed to obtain registration.
    throw new IOException("Not ready to serve the block pool, " + bpId + ".");
  }

  private void checkAccess(OutputStream out, final boolean reply,
      ExtendedBlock blk, Token<BlockTokenIdentifier> t, Op op,
      BlockTokenIdentifier.AccessMode mode) throws IOException {
    checkAccess(out, reply, blk, t, op, mode, null, null);
  }

  private void checkAccess(OutputStream out, final boolean reply,
      final ExtendedBlock blk,
      final Token<BlockTokenIdentifier> t,
      final Op op,
      final BlockTokenIdentifier.AccessMode mode,
      final StorageType[] storageTypes,

View on GitHub (pinned to 2add963021)

Solutions

1. Treat it as expected noise during a shutdown: verify the DataNode is indeed stopping (log shows shutdown sequence) and drain clients before restarting
2. For rolling operations, decommission/stop client traffic to that node first (e.g. exclude list or load balancer) so requests do not arrive mid-shutdown
3. If it appears while the DataNode should stay up, look for what interrupted the thread (OOM killer, monitoring scripts sending signals, container orchestrator liveness probes killing the JVM)
4. Retry the client read/write; HDFS clients retry transparently against other replicas, so no data is lost

// Client side: rely on built-in retries; DN-side this is shutdown noise, not a defect
try {
  in = dfsClient.open(path).read(...);
} catch (IOException e) {
  if (e.getMessage().contains("Interrupted while serving request")) {
    // DN is shutting down - the DFSClient fails over to another replica automatically;
    // simply retry once if you disabled retries
  }
}

When it happens

Trigger: A client request for a block pool reaches the DataNode before that block pool has registered; the xceiver enters the registration wait loop (up to dfs.datanode.bp-ready.timeout, default 20 seconds); during the sleep, DataNode shutdown or shutdownForUpgrade interrupts the thread, converting InterruptedException into this IOException.

Common situations: Rolling restarts or upgrades where clients keep sending reads while a DataNode process is being stopped; HA failover storms where block pools deregister and xceivers pile up, then the node is restarted; graceful shutdown with '-upgrade' or '-rollback' while requests are in flight.

Related errors

• Router Federation Rename is interrupted while checking permi
• Interrupted waiting " + timeoutMs + "ms for a quorum of node
• Interrupted waiting for format() response
• Interrupted while determining if JNs have data
• Interrupted waiting for doPreUpgrade() response

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3158eed49373d2f5. Report an issue: GitHub.