apache/hadoop · critical · PathIOException

Encoded SHA256 hash must be provided for global encryption

Error message

Encoded SHA256 hash must be provided for global encryption

What it means

Client-provided encryption keys (CPPK) require the service to verify key integrity via x-ms-encryption-key-sha256, so Hadoop demands both halves of the pair. At initialization, if fs.azure.encryption.encoded.client-provided-key is set on a secure connection but fs.azure.encryption.encoded.client-provided-key-sha is missing, PathIOException 'Encoded SHA256 hash must be provided for global encryption' aborts client construction.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:1891

          abfsConfiguration.getRawConfiguration());
    }

    // Encryption setup
    EncryptionContextProvider encryptionContextProvider = null;
    if (isSecure) {
      encryptionContextProvider =
          abfsConfiguration.createEncryptionContextProvider();
      if (encryptionContextProvider != null) {
        if (abfsConfiguration.getEncodedClientProvidedEncryptionKey() != null) {
          throw new PathIOException(uri.getPath(),
              "Both global key and encryption context are set, only one allowed");
        }
        encryptionContextProvider.initialize(
            abfsConfiguration.getRawConfiguration(), accountName,
            fileSystemName);
      } else if (abfsConfiguration.getEncodedClientProvidedEncryptionKey() != null) {
        if (abfsConfiguration.getEncodedClientProvidedEncryptionKeySHA() == null) {
          throw new PathIOException(uri.getPath(),
              "Encoded SHA256 hash must be provided for global encryption");
        }
      }
    }

    LOG.trace("Initializing AbfsClientHandler for {}", baseUrl);
    this.clientHandler = new AbfsClientHandler(baseUrl, creds,
        abfsConfiguration,
        tokenProvider, sasTokenProvider, encryptionContextProvider,
        populateAbfsClientContext());

    this.setClient(getClientHandler().getClient());
    LOG.trace("AbfsClient init complete");
  }

  private AbfsServiceType getAbfsServiceTypeFromUrl() {
    if (uri.toString().contains(ABFS_BLOB_DOMAIN_NAME)) {
      return AbfsServiceType.BLOB;

View on GitHub (pinned to 2add963021)

Solutions

Compute SHA-256 of the raw key bytes and add fs.azure.encryption.encoded.client-provided-key-sha=<base64 hash>.
Both values are Base64: key = base64(raw AES key), sha = base64(SHA256(raw key bytes)) - not SHA256 of the base64 string.
Check for account-scoped variants (fs.azure.account.<acct>.) overriding or shadowing the cluster-wide properties.
Remember CPPK applies only on secure (abfss/https) connections; on insecure endpoints the pair is ignored rather than validated.

Example fix

# compute the required sha (linux)
printf '%s' '<raw-key-bytes>' | openssl dgst -sha256 -binary | base64

<property>
  <name>fs.azure.encryption.encoded.client-provided-key-sha</name>
  <value>BASE64_OF_SHA256_OF_RAW_KEY</value>
</property>

Defensive patterns

Strategy: validation

Validate before calling

String key = conf.get("fs.azure.encryption.encoded.client-provided-key");
String sha = conf.get("fs.azure.encryption.encoded.client-provided-key-sha");
if (key != null && sha == null) {
  throw new IllegalArgumentException("fs.azure.encryption.encoded.client-provided-key-sha is required with the key");
}
// sanity: sha must decode to 32 bytes
if (sha != null && java.util.Base64.getDecoder().decode(sha).length != 32) {
  throw new IllegalArgumentException("-sha must be base64 of a 32-byte SHA-256");
}

Try / catch

try { fs = FileSystem.get(conf); } catch (PathIOException e) { if (e.getMessage().contains("SHA256")) { /* add the -sha property: base64(SHA256(raw key bytes)) */ } throw e; }

Prevention

Generate key and sha together in one script; store them as a pair.
sha = base64(SHA256(raw key bytes)), not a hash of the base64 string.
Decode-check the sha (must be 32 bytes) in config validation.

When it happens

Trigger: Secure abfss:// mount (no encryption-context provider) with fs.azure.encryption.encoded.client-provided-key configured but the matching fs.azure.encryption.encoded.client-provided-key-sha property absent or misspelled (e.g. missing the account-specific overlay).

Common situations: Key rotated but hash line forgotten; typo in the -sha property name; copy-paste from docs that show only the key; config management dropping the second property.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/71f100678b06e162. Report an issue: GitHub.