apache/hadoop · error · IOException

Failed to read block %s for file %s from datanode %s. Except

Error message

Failed to read block %s for file %s from datanode %s. Exception is %s. Retry with the next available datanode.

What it means

The per-DN read failure wrapper in fetchBlockByteRange: after token/encryption-key refetch attempts did not help, the client logs the failure, records the exception in exceptionMap keyed by the DN address, marks the DN dead locally and in the dead-node detector, and rethrows as IOException so readBuffer selects the next replica. The message (with the underlying exception stringified) only escapes to the caller when every replica fails.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1290

          try {
            fetchBlockAt(block.getStartOffset());
          } catch (IOException fbae) {
            // ignore IOE, since we can retry it later in a loop
          }
        } else {
          String msg = String.format("Failed to read block %s for file %s from datanode %s. "
                  + "Exception is %s. Retry with the next available datanode.",
              block.getBlock().getBlockName(), src, datanode.addr, e);
          DFSClient.LOG.warn(msg);

          // Add the exception to the exceptionMap
          if (!exceptionMap.containsKey(datanode.addr)) {
            exceptionMap.put(datanode.addr, new LinkedList<IOException>());
          }
          exceptionMap.get(datanode.addr).add(e);
          addToLocalDeadNodes(datanode.info);
          dfsClient.addNodeToDeadNodeDetector(this, datanode.info);
          throw new IOException(msg);
        }
        // Refresh the block for updated tokens in case of token failures or
        // encryption key failures.
        block = refreshLocatedBlock(block);
      } finally {
        if (reader != null) {
          reader.close();
        }
      }
    }
  }

  /**
   * Refresh cached block locations.
   * @param block The currently cached block locations
   * @return Refreshed block locations
   * @throws IOException
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Check the logged root exception in this message - it names the actual DN-side cause
2. For token expiry, ensure the client can re-fetch tokens (NN reachable, delegation tokens valid, no clock skew)
3. For encryption-key failures, let the client refetch the key (it retries once) and verify KMS connectivity
4. Fix the DN/network issues causing the underlying failures; the client's replica failover handles the rest

static boolean isDatanodeReadFailure(IOException e) {
  return e.getMessage() != null
      && e.getMessage().startsWith("Failed to read block");
}

for (int i = 0; i < 3; i++) {
  try { return readRange(in, off, len); }
  catch (IOException e) {
    if (i == 2 || !isDatanodeReadFailure(e)) throw e;
    Thread.sleep(1000L * (i + 1)); // replicas already rotated; give transient DN issues time
  }
}

When it happens

Trigger: DN read errors during fetchBlockByteRange: connection resets, DN restarts, block token expiry/refresh failures, InvalidEncryptionKeyException retries exhausted, disk errors on the DN.

Common situations: DN restarts or GC pauses under read load; network partitions between client and DNs; long-running readers outliving block token validity (especially after NN restart regenerating keys); encrypted (KMS/DFS encryption) zones with stale encryption keys.

Related errors

• Block pool {bpid} has not recognized an active NN
• Unexpected EOS from the reader
• ${className} does not support seekToNewSource.
• {} is in observer state. Cannot be failover target
• Version Mismatch (Expected: {}, Received: {} )

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a78bd64e1edc77c1. Report an issue: GitHub.