What causes "blockSize should be a multiple of checksumsize" in apache/hadoop?

Round the block size to a multiple of bytesPerChecksum (typically 512) before passing it — e.g. round up: bs = ((bs + bpc - 1) / bpc) * bpc

apache/hadoop · error · HadoopIllegalArgumentException

blockSize should be a multiple of checksumsize

Error message

blockSize should be a multiple of checksumsize

What it means

Final argument validation in AbstractFileSystem.create: the resolved blockSize (user option or server default) must be divisible by the resolved bytesPerChecksum (user option or the 512 default), else HadoopIllegalArgumentException. Unlike the server-defaults internal check, this validates the merged values the caller actually influenced.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:634

    // checksumOpt. Any missing value will be filled in using the default.
    ChecksumOpt defaultOpt = new ChecksumOpt(
        ssDef.getChecksumType(),
        ssDef.getBytesPerChecksum());
    checksumOpt = ChecksumOpt.processChecksumOpt(defaultOpt,
        checksumOpt, bytesPerChecksum);

    if (bufferSize == -1) {
      bufferSize = ssDef.getFileBufferSize();
    }
    if (replication == -1) {
      replication = ssDef.getReplication();
    }
    if (createParent == null) {
      createParent = false;
    }

    if (blockSize % bytesPerChecksum != 0) {
      throw new HadoopIllegalArgumentException(
             "blockSize should be a multiple of checksumsize");
    }

    return this.createInternal(f, createFlag, permission, bufferSize,
      replication, blockSize, progress, checksumOpt, createParent);
  }

  /**
   * The specification of this method matches that of
   * {@link #create(Path, EnumSet, Options.CreateOpts...)} except that the opts
   * have been declared explicitly.
   *
   * @param f the path.
   * @param flag create flag.
   * @param absolutePermission absolute permission.
   * @param bufferSize buffer size.
   * @param replication replications.
   * @param blockSize block size.

View on GitHub (pinned to 2add963021)

Solutions

Round the block size to a multiple of bytesPerChecksum (typically 512) before passing it — e.g. round up: bs = ((bs + bpc - 1) / bpc) * bpc
Omit CreateOpts.blockSize and use the server default, which is already consistent
When changing io.bytes.per.checksum cluster-side, audit code that passes explicit block sizes

Example fix

// before
fc.create(f, flag, CreateOpts.blockSize(1_000_000));

// after
long bpc = 512;
long bs = ((1_000_000 + bpc - 1) / bpc) * bpc; // 1000448
fc.create(f, flag, CreateOpts.blockSize(bs));

Defensive patterns

Strategy: validation

Validate before calling

long bpc = CreateOpts.getOpt(CreateOpts.BytesPerChecksum.class, opts) != null
    ? CreateOpts.getOpt(CreateOpts.BytesPerChecksum.class, opts).getValue() : 512;
if (blockSize % bpc != 0) {
  blockSize = ((blockSize + bpc - 1) / bpc) * bpc; // round up to a multiple
}
fc.create(f, flag, CreateOpts.blockSize(blockSize));

Prevention

Round computed block sizes to multiples of the checksum size before passing them
Omit blockSize and rely on server defaults unless tuning is required
When changing bytesPerChecksum, audit call sites that pass explicit block sizes

When it happens

Trigger: CreateOpts.blockSize(1_000_000) with the default 512-byte checksum; combining CreateOpts.blockSize(...) with CreateOpts.bytesPerChecksum(1024) where the former is not a multiple of the latter; computing block size from payload/quota sizes and passing it raw.

Common situations: Apps deriving blockSize from record counts or user settings; switching bytesPerChecksum for performance without adjusting block size; small test files created with tiny odd block sizes.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/609e2ed7b8ae1814. Report an issue: GitHub.