apache/hadoop · error · IOException

MD5 algorithm not available

Error message

MD5 algorithm not available

What it means

AbfsOutputStream's constructor creates MessageDigest.getInstance("MD5") digests for incremental and full-blob checksums. If the JVM's security providers do not supply MD5 (NoSuchAlgorithmException) AND fs.azure.enable.checksum.validation is true, it throws IOException("MD5 algorithm not available", e). With validation disabled (the default, DEFAULT_ENABLE_ABFS_CHECKSUM_VALIDATION = false) the missing digest is tolerated.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java:227

    this.outputStreamId = createOutputStreamId();
    this.tracingContext = new TracingContext(abfsOutputStreamContext.getTracingContext());
    this.tracingContext.setStreamID(outputStreamId);
    this.tracingContext.setOperation(FSOperationType.WRITE);
    this.ioStatistics = outputStreamStatistics.getIOStatistics();
    this.blockFactory = abfsOutputStreamContext.getBlockFactory();
    this.isDFSToBlobFallbackEnabled
        = abfsOutputStreamContext.isDFSToBlobFallbackEnabled();
    this.serviceTypeAtInit = abfsOutputStreamContext.getIngressServiceType();
    this.currentExecutingServiceType = abfsOutputStreamContext.getIngressServiceType();
    this.clientHandler = abfsOutputStreamContext.getClientHandler();
    createIngressHandler(serviceTypeAtInit,
        abfsOutputStreamContext.getBlockFactory(), bufferSize, false, null);
    try {
      md5 = MessageDigest.getInstance(MD5);
      fullBlobContentMd5 = MessageDigest.getInstance(MD5);
    } catch (NoSuchAlgorithmException e) {
      if (isChecksumValidationEnabled()) {
        throw new IOException("MD5 algorithm not available", e);
      }
    }
  }

  /**
   * Retrieves the current ingress handler.
   *
   * @return the current {@link AzureIngressHandler}.
   */
  public AzureIngressHandler getIngressHandler() {
    return ingressHandler;
  }

  private final Lock lock = new ReentrantLock();

  private volatile boolean switchCompleted = false;

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Set fs.azure.enable.checksum.validation to false (it is the default) unless MD5 validation is required
2. If validation is required, run on a JVM/provider set where MD5 is permitted (FIPS policy exception or non-FIPS JVM)
3. Verify with a quick probe: MessageDigest.getInstance("MD5") in the target JVM

Example fix

<!-- before -->
<property>
  <name>fs.azure.enable.checksum.validation</name>
  <value>true</value>
</property>

<!-- after -->
<property>
  <name>fs.azure.enable.checksum.validation</name>
  <value>false</value>
</property>

// Probe MD5 availability before enabling checksum validation
boolean md5Ok;
try {
  java.security.MessageDigest.getInstance("MD5");
  md5Ok = true;
} catch (java.security.NoSuchAlgorithmException e) {
  md5Ok = false;
}
conf.setBoolean("fs.azure.enable.checksum.validation",
    md5Ok && wantChecksumValidation);

When it happens

Trigger: Running on a FIPS 140-2/140-3-enforced JVM (e.g., BouncyCastle FIPS provider) where MD5 is disabled by policy; a custom java.security provider list that omits SUN/MessageDigest MD5; hardened container base images that strip algorithms; unusual JREs with restricted crypto policies.

Common situations: Regulated environments (government, finance, healthcare) booting Hadoop workers on FIPS mode; JVM upgrades that changed default security providers; Docker base images built with restricted java.security.

Related errors

• Lease desired but no lease threads configured, set fs.azure.
• - account name is not fully qualified.
• Exception while initializing metric credentials
• The ingress service type must be configured as DFS
• Internal error: default blockSize is not a multiple of defau

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/43a54b1958598922. Report an issue: GitHub.