apache/hadoop · error · UnsupportedCodecException

AES/CTR/NoPadding or SM4/CTR/NoPadding is required

Error message

AES/CTR/NoPadding or SM4/CTR/NoPadding is required

What it means

CryptoStreamUtils.checkCodec enforces that Hadoop transparent-encryption streams only use counter mode ciphers: AES/CTR/NoPadding or SM4/CTR/NoPadding. CTR mode is required because encrypted streams must preserve plaintext length and be seekable. Any codec whose CipherSuite is neither of these is rejected with UnsupportedCodecException.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java:78

   * Read crypto buffer size.
   *
   * @param conf configuration.
   * @return hadoop.security.crypto.buffer.size.
   */
  public static int getBufferSize(Configuration conf) {
    return conf.getInt(HADOOP_SECURITY_CRYPTO_BUFFER_SIZE_KEY, 
        HADOOP_SECURITY_CRYPTO_BUFFER_SIZE_DEFAULT);
  }

  /**
   * AES/CTR/NoPadding or SM4/CTR/NoPadding is required.
   *
   * @param codec crypto codec.
   */
  public static void checkCodec(CryptoCodec codec) {
    if (codec.getCipherSuite() != CipherSuite.AES_CTR_NOPADDING &&
            codec.getCipherSuite() != CipherSuite.SM4_CTR_NOPADDING) {
      throw new UnsupportedCodecException(
          "AES/CTR/NoPadding or SM4/CTR/NoPadding is required");
    }
  }

  /**
   * Check and floor buffer size.
   *
   * @param codec crypto codec.
   * @param bufferSize the size of the buffer to be used.
   * @return calc buffer size.
   */
  public static int checkBufferSize(CryptoCodec codec, int bufferSize) {
    Preconditions.checkArgument(bufferSize >= MIN_BUFFER_SIZE, 
        "Minimum value of buffer size is " + MIN_BUFFER_SIZE + ".");
    return bufferSize - bufferSize % codec.getCipherSuite()
        .getAlgorithmBlockSize();
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Set hadoop.security.crypto.cipher.suite to AES/CTR/NoPadding (the default and safest choice)
2. If SM4 is required, use SM4/CTR/NoPadding and confirm the native OpenSSL library supports it
3. For custom codecs, return CipherSuite.AES_CTR_NOPADDING or SM4_CTR_NOPADDING from getCipherSuite()
4. Remove or correct any client-side override of the cipher suite configuration

Example fix

// before
conf.set("hadoop.security.crypto.cipher.suite", "AES/CBC/PKCS5Padding");
CryptoCodec codec = CryptoCodec.getInstance(conf); // later throws in checkCodec

// after
conf.set("hadoop.security.crypto.cipher.suite", "AES/CTR/NoPadding");
CryptoCodec codec = CryptoCodec.getInstance(conf);

// Validate before use
CipherSuite suite = codec.getCipherSuite();
if (suite != CipherSuite.AES_CTR_NOPADDING && suite != CipherSuite.SM4_CTR_NOPADDING) {
  throw new IllegalArgumentException("Only CTR suites supported, got: " + suite);
}

public boolean isAcceptableSuite(CryptoCodec codec) {
  CipherSuite s = codec.getCipherSuite();
  return s == CipherSuite.AES_CTR_NOPADDING || s == CipherSuite.SM4_CTR_NOPADDING;
}

try {
  CryptoStreamUtils.checkCodec(codec);
} catch (UnsupportedCodecException e) {
  // fall back to default AES/CTR codec
  codec = CryptoCodec.getInstance(defaultConf);
}

When it happens

Trigger: Passing a CryptoCodec whose getCipherSuite() returns a suite other than AES_CTR_NOPADDING or SM4_CTR_NOPADDING into checkCodec — e.g. after setting hadoop.security.crypto.cipher.suite to a non-CTR value, or plugging in a custom CryptoCodec implementation with a different suite.

Common situations: Setting hadoop.security.crypto.cipher.suite to something like AES/CBC/PKCS5Padding or AES/GCM/NoPadding; developing a custom CryptoCodec and forgetting Hadoop only accepts CTR-mode suites; SM4 configured on a cluster where code paths still validate against AES-only expectations.

Related errors

• Invalid cipher suite name: ${name}
• The OpenSSL native library is built without SM4 CTR support
• No configuration found for the cipher suite {} prefixed with
• Invalid cipher suite, %s=%s
• Compression codec {} was not found.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6cb4722229a788d5. Report an issue: GitHub.