{"record":{"id":"bf6c42ea304f55d0","repo":"apache/hadoop","slug":"invalid-values-dfs-bytes-per-checksum-must","errorCode":null,"errorMessage":"Invalid values: dfs.bytes-per-checksum (={}) must divide block size (={}).","messagePattern":"Invalid values: dfs\\.bytes-per-checksum \\(=(.+?)\\) must divide block size \\(=(.+?)\\)\\.","errorType":"validation","errorClass":"HadoopIllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java","lineNumber":237,"sourceCode":"      this.addBlockFlags.add(AddBlockFlag.NO_LOCAL_RACK);\n    }\n    if (flag.contains(CreateFlag.IGNORE_CLIENT_LOCALITY)) {\n      this.addBlockFlags.add(AddBlockFlag.IGNORE_CLIENT_LOCALITY);\n    }\n    if (progress != null) {\n      DFSClient.LOG.debug(\"Set non-null progress callback on DFSOutputStream \"\n          +\"{}\", src);\n    }\n\n    initWritePacketSize();\n\n    this.bytesPerChecksum = checksum.getBytesPerChecksum();\n    if (bytesPerChecksum <= 0) {\n      throw new HadoopIllegalArgumentException(\n          \"Invalid value: bytesPerChecksum = \" + bytesPerChecksum + \" <= 0\");\n    }\n    if (blockSize % bytesPerChecksum != 0) {\n      throw new HadoopIllegalArgumentException(\"Invalid values: \"\n          + HdfsClientConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY\n          + \" (=\" + bytesPerChecksum + \") must divide block size (=\" +\n          blockSize + \").\");\n    }\n    this.byteArrayManager = dfsClient.getClientContext().getByteArrayManager();\n  }\n\n  /**\n   * Ensures the configured writePacketSize never exceeds\n   * PacketReceiver.MAX_PACKET_SIZE.\n   */\n  private void initWritePacketSize() {\n    writePacketSize = dfsClient.getConf().getWritePacketSize();\n    if (writePacketSize > PacketReceiver.MAX_PACKET_SIZE) {\n      LOG.warn(\n          \"Configured write packet exceeds {} bytes as max,\"\n              + \" using {} bytes.\",\n          PacketReceiver.MAX_PACKET_SIZE, PacketReceiver.MAX_PACKET_SIZE);","sourceCodeStart":219,"sourceCodeEnd":255,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L219-L255","documentation":"The DFSOutputStream constructor enforces blockSize % bytesPerChecksum == 0 and throws HadoopIllegalArgumentException otherwise. Every block must decompose into whole checksum chunks, so an odd pairing (e.g. blockSize 1000 with bytesPerChecksum 512) is rejected at stream creation, before any RPC. The two values arrive from dfs.blocksize / the create() blockSize argument and dfs.bytes-per-checksum / the ChecksumOpt.","triggerScenarios":"Calling fs.create(path, true, bufSize, replication, 1000L, null) with the default 512-byte checksum; a custom dfs.blocksize (e.g. 1048577) that is not a multiple of dfs.bytes-per-checksum; append when the client's current dfs.bytes-per-checksum differs from what the file was written with.","commonSituations":"Test clusters or benchmarks setting 'round' decimal block sizes (1000, 10000) that are not multiples of 512; applications hard-coding a blockSize while a site config overrides bytes-per-checksum to a non-power-of-two (e.g. 1000) after an upgrade; environments where different tools create and append with mismatched configs.","solutions":["Make the block size a multiple of the checksum chunk size - safest is to keep defaults (dfs.blocksize 128/256MB, dfs.bytes-per-checksum 512) or use powers of two.","Validate before creating: if (blockSize % bytesPerChecksum != 0) round blockSize up to the next multiple before passing it to create().","On append, do not override the checksum: pass null as checksumOpt so the file's existing layout wins.","Diff the client configuration (hadoop conf / -fs config dump) between the creating tool and the appending tool to find the divergent property."],"exampleFix":"// before\nFSDataOutputStream out = fs.create(path, true, 4096, (short) 3, 1000L, null);\n// 1000 % 512 != 0 -> HadoopIllegalArgumentException\n\n// after\nFSDataOutputStream out = fs.create(path, true, 4096, (short) 3, 1024L, null);\n// 1024 % 512 == 0 -> ok","handlingStrategy":"validation","validationCode":"int bytesPerChecksum = conf.getInt(\"dfs.bytes-per-checksum\", 512); // or checksumOpt\nlong blockSize = desiredBlockSize;\nif (blockSize % bytesPerChecksum != 0) {\n  blockSize = ((blockSize / bytesPerChecksum) + 1) * bytesPerChecksum; // round up\n}\nFSDataOutputStream out = fs.create(path, true, 4096, (short) 3, blockSize, null);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep defaults (256MB block / 512B checksum) unless a measured need says otherwise.","Any custom block size must be a multiple of the checksum chunk size - prefer powers of two.","On append pass null checksumOpt so the file's existing layout wins.","Diff create-side and append-side client configs for these two keys."],"tags":["hdfs","hdfs-client","config","block-size","checksum","validation"],"backgroundTag":"block-size-checksum-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}