apache/hadoop · error · IOException

Internal error: default blockSize is not a multiple of defau

Error message

Internal error: default blockSize is not a multiple of default bytesPerChecksum

What it means

Before resolving defaults, AbstractFileSystem.create fetches FsServerDefaults via getServerDefaults(f) and asserts server blockSize % bytesPerChecksum == 0; failure raises IOException("Internal error: ..."). It signals that the effective server-side configuration (or a custom file system's FsServerDefaults) is self-inconsistent — the message explicitly calls it an internal error, not a user-input error.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java:606

      } else if (CreateOpts.CreateParent.class.isInstance(iOpt)) {
        if (createParent != null) {
          throw new HadoopIllegalArgumentException(
              "CreateParent option is set multiple times");
        }
        createParent = ((CreateOpts.CreateParent) iOpt).getValue();
      } else {
        throw new HadoopIllegalArgumentException("Unkown CreateOpts of type " +
            iOpt.getClass().getName());
      }
    }
    if (permission == null) {
      throw new HadoopIllegalArgumentException("no permission supplied");
    }


    FsServerDefaults ssDef = getServerDefaults(f);
    if (ssDef.getBlockSize() % ssDef.getBytesPerChecksum() != 0) {
      throw new IOException("Internal error: default blockSize is" + 
          " not a multiple of default bytesPerChecksum ");
    }
    
    if (blockSize == -1) {
      blockSize = ssDef.getBlockSize();
    }

    // Create a checksum option honoring user input as much as possible.
    // If bytesPerChecksum is specified, it will override the one set in
    // checksumOpt. Any missing value will be filled in using the default.
    ChecksumOpt defaultOpt = new ChecksumOpt(
        ssDef.getChecksumType(),
        ssDef.getBytesPerChecksum());
    checksumOpt = ChecksumOpt.processChecksumOpt(defaultOpt,
        checksumOpt, bytesPerChecksum);

    if (bufferSize == -1) {
      bufferSize = ssDef.getFileBufferSize();

View on GitHub (pinned to 2add963021)

Solutions

Set dfs.block.size to a multiple of io.bytes.per.checksum (e.g. 134217728 = 128MB with 512-byte checksums)
Verify what getServerDefaults(path) actually returns for the target path/cluster and fix the source of the bad value
For custom file systems, build FsServerDefaults so blockSize is divisible by bytesPerChecksum (and sanity-check in tests)

Example fix

# before (core-site.xml / hdfs-site.xml)
<property><name>dfs.block.size</name><value>1000000</value></property>

# after
<property><name>dfs.block.size</name><value>134217728</value></property>

Defensive patterns

Strategy: validation

Validate before calling

FsServerDefaults d = afs.getServerDefaults(f);
if (d.getBlockSize() % d.getBytesPerChecksum() != 0) {
  throw new IllegalStateException("Server defaults inconsistent: blockSize=" + d.getBlockSize()
      + " bytesPerChecksum=" + d.getBytesPerChecksum() + " — fix dfs.block.size / io.bytes.per.checksum");
}

Try / catch

catch (IOException e) { if (e.getMessage().contains("not a multiple of default bytesPerChecksum")) { /* config error: surface to operator, do not retry */ } else throw e; }

Prevention

Keep dfs.block.size a power-of-two multiple of io.bytes.per.checksum (default 512)
For custom file systems, build FsServerDefaults from a consistent base and test defaults in CI
Log getServerDefaults() values when onboarding a new cluster or FS implementation

When it happens

Trigger: dfs.block.size configured to a value not divisible by io.bytes.per.checksum (default 512); a custom AbstractFileSystem/FileSystem returning a hand-built FsServerDefaults with inconsistent values; test stubs/fake file systems with arbitrary defaults.

Common situations: Operators tuning dfs.block.size to a round decimal (e.g. 1000000) that is not a multiple of 512; in-memory/minicluster setups with locally overridden defaults; third-party filesystem implementations with sloppy FsServerDefaults.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b3b29db6cb308b5b. Report an issue: GitHub.