apache/iceberg · error · RuntimeException

Failed to create message digest needed for s3 checksum…

Error message

Failed to create message digest needed for s3 checksum checks.

What it means

newStream() creates a per-part MessageDigest for multipart checksum tracking and throws a RuntimeException if the digest algorithm cannot be instantiated. Same root cause as the constructor variant but raised lazily when a new multipart part stream is opened.

Solutions

Fix the JVM's security provider configuration so MessageDigest.getInstance succeeds.
Run on a standard full JDK.
Disable s3.checksum-enabled if digest support is unavailable.
Check for JVM-wide security policy restrictions on JCE algorithms.

Example fix

// before
props.put("s3.checksum-enabled", "true"); // fails at part rollover
// after
props.put("s3.checksum-enabled", "false");

Defensive patterns

Strategy: validation

Validate before calling

// Java
try {
  MessageDigest.getInstance("SHA-256");
} catch (NoSuchAlgorithmException e) {
  props.setChecksumEnabled(false); // avoid mid-write failure at part rollover
}

Try / catch

// Java
try {
  out.write(largeBuffer); // triggers newStream at part boundary
} catch (RuntimeException e) {
  LOG.error("Digest init failed mid-write: {}", e.getCause());
  throw e;
}

Prevention

Validate digest availability at job startup, not lazily
Use standard JDK distributions on executor nodes
Avoid custom security policies that restrict JCE algorithms

When it happens

Trigger: Writing enough data to roll into a new multipart part while s3.checksum-enabled=true and the JVM cannot supply the digest algorithm.

Common situations: Same minimal/stripped JRE or broken security-provider environments as error 363; surfaces mid-write instead of at stream construction.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d686a17458fe9303. Report an issue: GitHub.

Appendix: source

Thrown at aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java:224

    // switch to multipart upload
    if (multipartUploadId == null && pos >= multiPartThresholdSize) {
      initializeMultiPartUpload();
      uploadParts();
    }
  }

  private void newStream() throws IOException {
    if (stream != null) {
      stream.close();
    }

    createStagingDirectoryIfNotExists();
    currentStagingFile = File.createTempFile("s3fileio-", ".tmp", stagingDirectory);
    try {
      currentPartMessageDigest =
          isChecksumEnabled ? MessageDigest.getInstance(DIGEST_ALGORITHM) : null;
    } catch (NoSuchAlgorithmException e) {
      throw new RuntimeException(
          "Failed to create message digest needed for s3 checksum checks.", e);
    }

    stagingFiles.add(new FileAndDigest(currentStagingFile, currentPartMessageDigest));
    OutputStream outputStream = Files.newOutputStream(currentStagingFile.toPath());

    if (isChecksumEnabled) {
      DigestOutputStream digestOutputStream;

      // if switched over to multipart threshold already, no need to update complete message digest
      if (multipartUploadId != null) {
        digestOutputStream =
            new DigestOutputStream(
                new BufferedOutputStream(outputStream), currentPartMessageDigest);
      } else {
        digestOutputStream =
            new DigestOutputStream(
                new DigestOutputStream(

View on GitHub (pinned to 86d9c8fc54)