apache/hadoop · error · HadoopIllegalArgumentException

Invalid values: dfs.bytes-per-checksum (={}) must divide blo

Error message

Invalid values: dfs.bytes-per-checksum (={}) must divide block size (={}).

What it means

The DFSOutputStream constructor enforces blockSize % bytesPerChecksum == 0 and throws HadoopIllegalArgumentException otherwise. Every block must decompose into whole checksum chunks, so an odd pairing (e.g. blockSize 1000 with bytesPerChecksum 512) is rejected at stream creation, before any RPC. The two values arrive from dfs.blocksize / the create() blockSize argument and dfs.bytes-per-checksum / the ChecksumOpt.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:237

      this.addBlockFlags.add(AddBlockFlag.NO_LOCAL_RACK);
    }
    if (flag.contains(CreateFlag.IGNORE_CLIENT_LOCALITY)) {
      this.addBlockFlags.add(AddBlockFlag.IGNORE_CLIENT_LOCALITY);
    }
    if (progress != null) {
      DFSClient.LOG.debug("Set non-null progress callback on DFSOutputStream "
          +"{}", src);
    }

    initWritePacketSize();

    this.bytesPerChecksum = checksum.getBytesPerChecksum();
    if (bytesPerChecksum <= 0) {
      throw new HadoopIllegalArgumentException(
          "Invalid value: bytesPerChecksum = " + bytesPerChecksum + " <= 0");
    }
    if (blockSize % bytesPerChecksum != 0) {
      throw new HadoopIllegalArgumentException("Invalid values: "
          + HdfsClientConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY
          + " (=" + bytesPerChecksum + ") must divide block size (=" +
          blockSize + ").");
    }
    this.byteArrayManager = dfsClient.getClientContext().getByteArrayManager();
  }

  /**
   * Ensures the configured writePacketSize never exceeds
   * PacketReceiver.MAX_PACKET_SIZE.
   */
  private void initWritePacketSize() {
    writePacketSize = dfsClient.getConf().getWritePacketSize();
    if (writePacketSize > PacketReceiver.MAX_PACKET_SIZE) {
      LOG.warn(
          "Configured write packet exceeds {} bytes as max,"
              + " using {} bytes.",
          PacketReceiver.MAX_PACKET_SIZE, PacketReceiver.MAX_PACKET_SIZE);

View on GitHub (pinned to 2add963021)

Solutions

Make the block size a multiple of the checksum chunk size - safest is to keep defaults (dfs.blocksize 128/256MB, dfs.bytes-per-checksum 512) or use powers of two.
Validate before creating: if (blockSize % bytesPerChecksum != 0) round blockSize up to the next multiple before passing it to create().
On append, do not override the checksum: pass null as checksumOpt so the file's existing layout wins.
Diff the client configuration (hadoop conf / -fs config dump) between the creating tool and the appending tool to find the divergent property.

Example fix

// before
FSDataOutputStream out = fs.create(path, true, 4096, (short) 3, 1000L, null);
// 1000 % 512 != 0 -> HadoopIllegalArgumentException

// after
FSDataOutputStream out = fs.create(path, true, 4096, (short) 3, 1024L, null);
// 1024 % 512 == 0 -> ok

Defensive patterns

Strategy: validation

Validate before calling

int bytesPerChecksum = conf.getInt("dfs.bytes-per-checksum", 512); // or checksumOpt
long blockSize = desiredBlockSize;
if (blockSize % bytesPerChecksum != 0) {
  blockSize = ((blockSize / bytesPerChecksum) + 1) * bytesPerChecksum; // round up
}
FSDataOutputStream out = fs.create(path, true, 4096, (short) 3, blockSize, null);

Prevention

Keep defaults (256MB block / 512B checksum) unless a measured need says otherwise.
Any custom block size must be a multiple of the checksum chunk size - prefer powers of two.
On append pass null checksumOpt so the file's existing layout wins.
Diff create-side and append-side client configs for these two keys.

When it happens

Trigger: Calling fs.create(path, true, bufSize, replication, 1000L, null) with the default 512-byte checksum; a custom dfs.blocksize (e.g. 1048577) that is not a multiple of dfs.bytes-per-checksum; append when the client's current dfs.bytes-per-checksum differs from what the file was written with.

Common situations: Test clusters or benchmarks setting 'round' decimal block sizes (1000, 10000) that are not multiples of 512; applications hard-coding a blockSize while a site config overrides bytes-per-checksum to a non-power-of-two (e.g. 1000) after an upgrade; environments where different tools create and append with mismatched configs.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bf6c42ea304f55d0. Report an issue: GitHub.