{"record":{"id":"98b4732fdc520443","repo":"apache/hadoop","slug":"invalid-values-dfs-bytes-per-checksum-must-98b473","errorCode":null,"errorMessage":"Invalid values: dfs.bytes-per-checksum (={}) must divide cell size (={}).","messagePattern":"Invalid values: dfs\\.bytes-per-checksum \\(=(.+?)\\) must divide cell size \\(=(.+?)\\)\\.","errorType":"exception","errorClass":"HadoopIllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java","lineNumber":212,"sourceCode":"      assert !updateStreamerMap.containsKey(streamer);\n      updateStreamerMap.put(streamer, success);\n    }\n\n    void clearFailureStates() {\n      newBlocks.clear();\n      updateStreamerMap.clear();\n      streamerUpdateResult.clear();\n    }\n  }\n\n  /** Buffers for writing the data and parity cells of a stripe. */\n  class CellBuffers {\n    private final ByteBuffer[] buffers;\n    private final byte[][] checksumArrays;\n\n    CellBuffers(int numParityBlocks) {\n      if (cellSize % bytesPerChecksum != 0) {\n        throw new HadoopIllegalArgumentException(\"Invalid values: \"\n            + HdfsClientConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY + \" (=\"\n            + bytesPerChecksum + \") must divide cell size (=\" + cellSize + \").\");\n      }\n\n      checksumArrays = new byte[numParityBlocks][];\n      final int size = getChecksumSize() * (cellSize / bytesPerChecksum);\n      for (int i = 0; i < checksumArrays.length; i++) {\n        checksumArrays[i] = new byte[size];\n      }\n\n      buffers = new ByteBuffer[numAllBlocks];\n      for (int i = 0; i < buffers.length; i++) {\n        buffers[i] = BUFFER_POOL.getBuffer(useDirectBuffer(), cellSize);\n        buffers[i].limit(cellSize);\n      }\n    }\n\n    private ByteBuffer[] getBuffers() {","sourceCodeStart":194,"sourceCodeEnd":230,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java#L194-L230","documentation":"When a write to an erasure-coded file starts, DFSStripedOutputStream builds CellBuffers — per-block data and parity cell buffers with checksum arrays. Its constructor requires that dfs.bytes-per-checksum divides the EC policy's cell size evenly (cellSize % bytesPerChecksum == 0), because checksums are laid out per cell chunk. A violation throws HadoopIllegalArgumentException naming both values. The check runs in the DFSStripedOutputStream constructor, i.e., at FileSystem.create()/append() time, before any bytes are written.","triggerScenarios":"Creating or appending an EC-coded file while dfs.bytes-per-checksum (client config, default 512) does not divide the policy cell size — e.g., RS-6-3-1024k (cell 1048576) with dfs.bytes-per-checksum=1000. All stock policies use cell sizes that are multiples of 1024, so default or power-of-two checksum sizes never trip this.","commonSituations":"Tuning dfs.bytes-per-checksum to non-power-of-two values carried over from another system; custom EC policies with odd cell sizes; client-side core-site.xml or Configuration.setLong diverging from cluster defaults (this validation uses the client's configuration).","solutions":["Set dfs.bytes-per-checksum back to the default 512 (or any divisor of the policy's cell size, e.g., 1024).","Or choose/define the EC policy so its cell size is a multiple of the configured checksum size.","Pre-validate in code: erasureCodingPolicy.getCellSize() % conf.getLong(DFS_BYTES_PER_CHECKSUM_KEY, 512) == 0 before creating the output stream.","Remember the Configuration is read when the FileSystem/DFSClient is created — apply the fix and recreate the client/FileSystem rather than mutating conf of a cached instance."],"exampleFix":"<!-- before: client core-site.xml -->\n<property><name>dfs.bytes-per-checksum</name><value>1000</value></property>\n\n<!-- after -->\n<property><name>dfs.bytes-per-checksum</name><value>512</value></property>","handlingStrategy":"validation","validationCode":"ErasureCodingPolicy ecPolicy =\n    fs.getClient().getErasureCodingPolicy(new Path(\"/ec-dir\"));\nif (ecPolicy != null) {\n  long bytesPerChecksum =\n      conf.getLong(DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY,\n                   DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_DEFAULT);\n  if (ecPolicy.getCellSize() % bytesPerChecksum != 0) {\n    throw new IllegalStateException(\"dfs.bytes-per-checksum (\" + bytesPerChecksum\n        + \") must divide EC cell size (\" + ecPolicy.getCellSize() + \")\");\n  }\n}","typeGuard":null,"tryCatchPattern":"try (FSDataOutputStream out = fs.create(path)) {\n  ...\n} catch (HadoopIllegalArgumentException e) {\n  // config mismatch: fix dfs.bytes-per-checksum or the EC policy, then retry\n}","preventionTips":["Keep dfs.bytes-per-checksum at the default 512 unless you also verify divisibility with every EC policy in use.","Add a startup config check that validates the divisibility invariant for all dir-level EC policies the app writes to.","Remember this validation uses the CLIENT's configuration — test with the same core-site.xml the app ships."],"tags":["hdfs","erasure-coding","configuration","checksum","validation","write"],"backgroundTag":"config-validation-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}