{"record":{"id":"ff07730798accb57","repo":"apache/hadoop","slug":"configured-blockreaderlocallegacy-buffer-size","errorCode":null,"errorMessage":"Configured BlockReaderLocalLegacy buffer size ({}) is not large enough to hold a single chunk ({}). Please configure dfs.client.read.shortcircuit.buffer.size appropriately","messagePattern":"Configured BlockReaderLocalLegacy buffer size \\((.+?)\\) is not large enough to hold a single chunk \\((.+?)\\)\\. Please configure dfs\\.client\\.read\\.shortcircuit\\.buffer\\.size appropriately","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderLocalLegacy.java","lineNumber":307,"sourceCode":"      // short-circuit read for all subsequent use in the ClientContext.  Unlike\n      // the newer short-circuit read implementation, we have no communication\n      // channel for the DataNode to notify the client that the path has been\n      // invalidated.  Therefore, our only option is to skip caching.\n      if (pathinfo != null && !storageType.isTransient()) {\n        LOG.debug(\"Cached location of block {} as {}\", blk, pathinfo);\n        localDatanodeInfo.setBlockLocalPathInfo(blk, pathinfo);\n      }\n    } catch (IOException e) {\n      localDatanodeInfo.resetDatanodeProxy(); // Reset proxy on error\n      throw e;\n    }\n    return pathinfo;\n  }\n\n  private static int getSlowReadBufferNumChunks(int bufferSizeBytes,\n      int bytesPerChecksum) {\n    if (bufferSizeBytes < bytesPerChecksum) {\n      throw new IllegalArgumentException(\"Configured BlockReaderLocalLegacy \" +\n          \"buffer size (\" + bufferSizeBytes + \") is not large enough to hold \" +\n          \"a single chunk (\" + bytesPerChecksum +  \"). Please configure \" +\n          HdfsClientConfigKeys.Read.ShortCircuit.BUFFER_SIZE_KEY +\n          \" appropriately\");\n    }\n\n    // Round down to nearest chunk size\n    return bufferSizeBytes / bytesPerChecksum;\n  }\n\n  private BlockReaderLocalLegacy(ShortCircuitConf conf, String hdfsfile,\n      ExtendedBlock block, long startOffset, FileInputStream dataIn)\n      throws IOException {\n    this(conf, hdfsfile, block, startOffset,\n        DataChecksum.newDataChecksum(DataChecksum.Type.NULL, 4), false,\n        dataIn, startOffset, null);\n  }\n","sourceCodeStart":289,"sourceCodeEnd":325,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderLocalLegacy.java#L289-L325","documentation":"The legacy short-circuit reader (BlockReaderLocalLegacy, used for local reads without a domain socket) sizes its read buffer in whole checksum chunks: getSlowReadBufferNumChunks throws IllegalArgumentException when dfs.client.read.shortcircuit.buffer.size (default 1 MiB) is smaller than the file's bytesPerChecksum, because the buffer could not hold even one chunk.","triggerScenarios":"Reading a file whose bytes-per-checksum exceeds dfs.client.read.shortcircuit.buffer.size — the buffer was tuned down (e.g., 256 bytes) while the file uses default or larger chunks, or files written with a large per-file bytesPerChecksum are read by clients with a shrunken buffer.","commonSituations":"Operators lower dfs.client.read.shortcircuit.buffer.size believing it is a small I/O buffer (it is a chunk buffer, default 1048576); mixed clusters where files created with non-default checksum settings are read by all clients.","solutions":["Restore or raise dfs.client.read.shortcircuit.buffer.size to at least the largest bytesPerChecksum in use; keeping the 1 MiB default is simplest.","Check the file's checksum layout (how it was created) to learn its bytesPerChecksum.","Prefer the modern short-circuit path (configure dfs.domain.socket.path), which does not use this legacy buffer math."],"exampleFix":"# before\n<property><name>dfs.client.read.shortcircuit.buffer.size</name><value>256</value></property>\n\n# after: at least bytesPerChecksum; 1 MiB default is safe\n<property><name>dfs.client.read.shortcircuit.buffer.size</name><value>1048576</value></property>","handlingStrategy":"validation","validationCode":"int buf = conf.getInt(\"dfs.client.read.shortcircuit.buffer.size\",\n    1024 * 1024);\nint bpc = conf.getInt(\"dfs.bytes-per-checksum\", 512);\nPreconditions.checkArgument(buf >= bpc,\n    \"dfs.client.read.shortcircuit.buffer.size (%s) must be >= \"\n    + \"bytesPerChecksum (%s)\", buf, bpc);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Fail fast on this config invariant at client startup, not on first read.","Do not shrink dfs.client.read.shortcircuit.buffer.size — the 1 MiB default covers normal chunk sizes.","When files are created with non-default bytesPerChecksum, verify every reader's buffer size covers it."],"tags":["hdfs","short-circuit-read","configuration","buffer-size","checksums"],"backgroundTag":"buffer-size-too-small","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}