apache/hadoop · error · HadoopIllegalArgumentException

Invalid value: bytesPerChecksum = {} <= 0

Error message

Invalid value: bytesPerChecksum = {} <= 0

What it means

When a DFSOutputStream is constructed (create or append), it reads bytesPerChecksum from the DataChecksum in effect and rejects values <= 0 with HadoopIllegalArgumentException. The value comes from dfs.bytes-per-checksum (default 512) or from an explicit ChecksumOpt passed to the create/append call. HDFS has no 'checksums off via zero' mode, so 0 or negative is always invalid configuration.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:233

    if (flag.contains(CreateFlag.NO_LOCAL_WRITE)) {
      this.addBlockFlags.add(AddBlockFlag.NO_LOCAL_WRITE);
    }
    if (flag.contains(CreateFlag.NO_LOCAL_RACK)) {
      this.addBlockFlags.add(AddBlockFlag.NO_LOCAL_RACK);
    }
    if (flag.contains(CreateFlag.IGNORE_CLIENT_LOCALITY)) {
      this.addBlockFlags.add(AddBlockFlag.IGNORE_CLIENT_LOCALITY);
    }
    if (progress != null) {
      DFSClient.LOG.debug("Set non-null progress callback on DFSOutputStream "
          +"{}", src);
    }

    initWritePacketSize();

    this.bytesPerChecksum = checksum.getBytesPerChecksum();
    if (bytesPerChecksum <= 0) {
      throw new HadoopIllegalArgumentException(
          "Invalid value: bytesPerChecksum = " + bytesPerChecksum + " <= 0");
    }
    if (blockSize % bytesPerChecksum != 0) {
      throw new HadoopIllegalArgumentException("Invalid values: "
          + HdfsClientConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY
          + " (=" + bytesPerChecksum + ") must divide block size (=" +
          blockSize + ").");
    }
    this.byteArrayManager = dfsClient.getClientContext().getByteArrayManager();
  }

  /**
   * Ensures the configured writePacketSize never exceeds
   * PacketReceiver.MAX_PACKET_SIZE.
   */
  private void initWritePacketSize() {
    writePacketSize = dfsClient.getConf().getWritePacketSize();
    if (writePacketSize > PacketReceiver.MAX_PACKET_SIZE) {

View on GitHub (pinned to 2add963021)

Solutions

1. Set dfs.bytes-per-checksum back to a sane positive value (512 is the default and standard).
2. If checksums genuinely must be disabled for a file, request the NULL checksum through the API (ChecksumOpt with DataChecksum.Type.NULL) rather than zeroing the size property - the size must still be > 0.
3. Validate any programmatically-built ChecksumOpt before calling create: assert opt.getBytesPerChecksum() > 0.
4. Check both client-side and server-side configs - append picks up the server's value, so a bad NN config hits even clean clients.

Example fix

// before
conf.setInt("dfs.bytes-per-checksum", 0); // hoping to disable checksums
FileSystem fs = FileSystem.get(conf);

// after
conf.unset("dfs.bytes-per-checksum"); // fall back to default 512
// to disable checksums per file, use the API instead:
// ((DistributedFileSystem) fs).create(path, ..., new ChecksumOpt(DataChecksum.Type.NULL, 512), ...);

int bpc = checksumOpt != null ? checksumOpt.getBytesPerChecksum() : 512;
if (bpc <= 0) throw new IllegalArgumentException("bytesPerChecksum must be > 0, got " + bpc);
FSDataOutputStream out = fs.create(path, true, 4096, (short) 3, 128L * 1024 * 1024, checksumOpt);

When it happens

Trigger: Setting dfs.bytes-per-checksum to 0 or negative in the client or server config; passing new ChecksumOpt(DataChecksum.Type.CRC32C, 0) (or a computed value that underflows) to DistributedFileSystem.create(..., checksumOpt, ...); append inheriting a server-side value that was misconfigured.

Common situations: Copied config templates with dfs.bytes-per-checksum=0 intended to 'disable' checksumming (not supported - per-file NULL checksum must be requested via API, not by zeroing this key); property typos like 512 accidentally edited to 51 or -512; code computing bytesPerChecksum from user input without validation.

Related errors

• Invalid values: dfs.bytes-per-checksum (={}) must divide blo
• Invalid value configured for dfs.datanode.disk.check.timeout
• Invalid value configured for dfs.datanode.disk.check.min.gap
• Invalid value configured for dfs.datanode.disk.check.timeout
• can't read a negative number of bytes.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b32d7de7e6daff57. Report an issue: GitHub.