{"record":{"id":"a78bd64e1edc77c1","repo":"apache/hadoop","slug":"failed-to-read-block-s-for-file-s-from-datanode","errorCode":null,"errorMessage":"Failed to read block %s for file %s from datanode %s. Exception is %s. Retry with the next available datanode.","messagePattern":"Failed to read block (.+?) for file (.+?) from datanode (.+?)\\. Exception is (.+?)\\. Retry with the next available datanode\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":1290,"sourceCode":"          try {\n            fetchBlockAt(block.getStartOffset());\n          } catch (IOException fbae) {\n            // ignore IOE, since we can retry it later in a loop\n          }\n        } else {\n          String msg = String.format(\"Failed to read block %s for file %s from datanode %s. \"\n                  + \"Exception is %s. Retry with the next available datanode.\",\n              block.getBlock().getBlockName(), src, datanode.addr, e);\n          DFSClient.LOG.warn(msg);\n\n          // Add the exception to the exceptionMap\n          if (!exceptionMap.containsKey(datanode.addr)) {\n            exceptionMap.put(datanode.addr, new LinkedList<IOException>());\n          }\n          exceptionMap.get(datanode.addr).add(e);\n          addToLocalDeadNodes(datanode.info);\n          dfsClient.addNodeToDeadNodeDetector(this, datanode.info);\n          throw new IOException(msg);\n        }\n        // Refresh the block for updated tokens in case of token failures or\n        // encryption key failures.\n        block = refreshLocatedBlock(block);\n      } finally {\n        if (reader != null) {\n          reader.close();\n        }\n      }\n    }\n  }\n\n  /**\n   * Refresh cached block locations.\n   * @param block The currently cached block locations\n   * @return Refreshed block locations\n   * @throws IOException\n   */","sourceCodeStart":1272,"sourceCodeEnd":1308,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1272-L1308","documentation":"The per-DN read failure wrapper in fetchBlockByteRange: after token/encryption-key refetch attempts did not help, the client logs the failure, records the exception in exceptionMap keyed by the DN address, marks the DN dead locally and in the dead-node detector, and rethrows as IOException so readBuffer selects the next replica. The message (with the underlying exception stringified) only escapes to the caller when every replica fails.","triggerScenarios":"DN read errors during fetchBlockByteRange: connection resets, DN restarts, block token expiry/refresh failures, InvalidEncryptionKeyException retries exhausted, disk errors on the DN.","commonSituations":"DN restarts or GC pauses under read load; network partitions between client and DNs; long-running readers outliving block token validity (especially after NN restart regenerating keys); encrypted (KMS/DFS encryption) zones with stale encryption keys.","solutions":["Check the logged root exception in this message - it names the actual DN-side cause","For token expiry, ensure the client can re-fetch tokens (NN reachable, delegation tokens valid, no clock skew)","For encryption-key failures, let the client refetch the key (it retries once) and verify KMS connectivity","Fix the DN/network issues causing the underlying failures; the client's replica failover handles the rest"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":"static boolean isDatanodeReadFailure(IOException e) {\n  return e.getMessage() != null\n      && e.getMessage().startsWith(\"Failed to read block\");\n}","tryCatchPattern":"for (int i = 0; i < 3; i++) {\n  try { return readRange(in, off, len); }\n  catch (IOException e) {\n    if (i == 2 || !isDatanodeReadFailure(e)) throw e;\n    Thread.sleep(1000L * (i + 1)); // replicas already rotated; give transient DN issues time\n  }\n}","preventionTips":["Read the embedded root exception string; it identifies the DN-side cause (token, network, disk)","Keep block tokens and encryption keys refreshable: NN reachable, clocks in sync, KMS up","Monitor DN liveness so failover has healthy replicas to choose"],"tags":["hdfs","datanode","failover","read"],"backgroundTag":"datanode-read-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}