apache/hadoop · error · IOException

Error generating encrypted spill key

Error message

Error generating encrypted spill key

What it means

When mapreduce.job.encrypted-intermediate-data=true, LocalJobRunner generates a spill-encryption key with KeyGenerator.getInstance("HmacSHA1") (INTERMEDIATE_DATA_ENCRYPTION_ALGO, LocalJobRunner.java:93; size from mapreduce.job.encrypted-intermediate-data.key.size.bits). If the JVM's security providers cannot supply HmacSHA1, NoSuchAlgorithmException is wrapped into this IOException and the local job aborts at start.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java:214

      jobs.put(id, this);

      if (CryptoUtils.isEncryptedSpillEnabled(job)) {
        try {
          int keyLen = conf.getInt(
              MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS,
              MRJobConfig
                  .DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS);
          KeyGenerator keyGen =
              KeyGenerator.getInstance(INTERMEDIATE_DATA_ENCRYPTION_ALGO);
          keyGen.init(keyLen);
          Credentials creds =
              UserGroupInformation.getCurrentUser().getCredentials();
          TokenCache.setEncryptedSpillKey(keyGen.generateKey().getEncoded(),
              creds);
          UserGroupInformation.getCurrentUser().addCredentials(creds);
        } catch (NoSuchAlgorithmException e) {
          throw new IOException("Error generating encrypted spill key", e);
        }
      }

      this.start();
    }

    protected abstract class RunnableWithThrowable implements Runnable {
      public volatile Throwable storedException;
    }

    /**
     * A Runnable instance that handles a map task to be run by an executor.
     */
    protected class MapTaskRunnable extends RunnableWithThrowable {
      private final int taskId;
      private final TaskSplitMetaInfo info;
      private final JobID jobId;
      private final JobConf localConf;

View on GitHub (pinned to 2add963021)

Solutions

1. Set mapreduce.job.encrypted-intermediate-data=false for local/test runs - spill encryption targets cluster intermediate data
2. Probe the JVM: KeyGenerator.getInstance("HmacSHA1") must succeed; on stock OpenJDK/Oracle 8+ it always does
3. Restore default security providers in java.security (SunJCE present, not commented out) or run with a standard JDK

Example fix

// before: local test job with spill encryption on
conf.setBoolean("mapreduce.job.encrypted-intermediate-data", true);
Job.getInstance(conf).submit(); // IOException at local job start

// after: encryption off for local runs
conf.setBoolean("mapreduce.job.encrypted-intermediate-data", false);
Job.getInstance(conf).submit();

// Probe the JVM before enabling spill encryption in local runs
try {
  javax.crypto.KeyGenerator.getInstance("HmacSHA1");
} catch (java.security.NoSuchAlgorithmException e) {
  conf.setBoolean("mapreduce.job.encrypted-intermediate-data", false);
  // or fail fast with a clear message about the JVM's security providers
}

try {
  Job.getInstance(conf).submit();
} catch (IOException io) {
  if (io.getCause() instanceof java.security.NoSuchAlgorithmException) {
    // JVM lacks HmacSHA1: disable local spill encryption or fix providers
    conf.setBoolean("mapreduce.job.encrypted-intermediate-data", false);
  } else {
    throw io;
  }
}

When it happens

Trigger: Running a local job with intermediate-data encryption enabled on a JVM whose provider list was modified - custom java.security, FIPS/BouncyCastle-only configurations, stripped or non-standard JRE builds - so the default Sun JCE provider with HmacSHA1 is absent.

Common situations: Corporate FIPS-hardened JVMs; minimal or split JRE container images; test JVMs that globally install a limited provider set; hand-edited java.security files after JDK upgrades.

Related errors

• Resource type PATTERN is not implemented yet. {}
• Configured mapreduce.local.map.tasks.maximum must be >= 1
• Configured mapreduce.local.reduce.tasks.maximum must be >= 1
• Verification of the hashReply failed
• security validation of TT Map output failed

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/75064ecb70c6d55a. Report an issue: GitHub.