{"record":{"id":"75064ecb70c6d55a","repo":"apache/hadoop","slug":"error-generating-encrypted-spill-key","errorCode":null,"errorMessage":"Error generating encrypted spill key","messagePattern":"Error generating encrypted spill key","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java","lineNumber":214,"sourceCode":"\n      jobs.put(id, this);\n\n      if (CryptoUtils.isEncryptedSpillEnabled(job)) {\n        try {\n          int keyLen = conf.getInt(\n              MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS,\n              MRJobConfig\n                  .DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS);\n          KeyGenerator keyGen =\n              KeyGenerator.getInstance(INTERMEDIATE_DATA_ENCRYPTION_ALGO);\n          keyGen.init(keyLen);\n          Credentials creds =\n              UserGroupInformation.getCurrentUser().getCredentials();\n          TokenCache.setEncryptedSpillKey(keyGen.generateKey().getEncoded(),\n              creds);\n          UserGroupInformation.getCurrentUser().addCredentials(creds);\n        } catch (NoSuchAlgorithmException e) {\n          throw new IOException(\"Error generating encrypted spill key\", e);\n        }\n      }\n\n      this.start();\n    }\n\n    protected abstract class RunnableWithThrowable implements Runnable {\n      public volatile Throwable storedException;\n    }\n\n    /**\n     * A Runnable instance that handles a map task to be run by an executor.\n     */\n    protected class MapTaskRunnable extends RunnableWithThrowable {\n      private final int taskId;\n      private final TaskSplitMetaInfo info;\n      private final JobID jobId;\n      private final JobConf localConf;","sourceCodeStart":196,"sourceCodeEnd":232,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java#L196-L232","documentation":"When mapreduce.job.encrypted-intermediate-data=true, LocalJobRunner generates a spill-encryption key with KeyGenerator.getInstance(\"HmacSHA1\") (INTERMEDIATE_DATA_ENCRYPTION_ALGO, LocalJobRunner.java:93; size from mapreduce.job.encrypted-intermediate-data.key.size.bits). If the JVM's security providers cannot supply HmacSHA1, NoSuchAlgorithmException is wrapped into this IOException and the local job aborts at start.","triggerScenarios":"Running a local job with intermediate-data encryption enabled on a JVM whose provider list was modified - custom java.security, FIPS/BouncyCastle-only configurations, stripped or non-standard JRE builds - so the default Sun JCE provider with HmacSHA1 is absent.","commonSituations":"Corporate FIPS-hardened JVMs; minimal or split JRE container images; test JVMs that globally install a limited provider set; hand-edited java.security files after JDK upgrades.","solutions":["Set mapreduce.job.encrypted-intermediate-data=false for local/test runs - spill encryption targets cluster intermediate data","Probe the JVM: KeyGenerator.getInstance(\"HmacSHA1\") must succeed; on stock OpenJDK/Oracle 8+ it always does","Restore default security providers in java.security (SunJCE present, not commented out) or run with a standard JDK"],"exampleFix":"// before: local test job with spill encryption on\nconf.setBoolean(\"mapreduce.job.encrypted-intermediate-data\", true);\nJob.getInstance(conf).submit(); // IOException at local job start\n\n// after: encryption off for local runs\nconf.setBoolean(\"mapreduce.job.encrypted-intermediate-data\", false);\nJob.getInstance(conf).submit();","handlingStrategy":"try-catch","validationCode":"// Probe the JVM before enabling spill encryption in local runs\ntry {\n  javax.crypto.KeyGenerator.getInstance(\"HmacSHA1\");\n} catch (java.security.NoSuchAlgorithmException e) {\n  conf.setBoolean(\"mapreduce.job.encrypted-intermediate-data\", false);\n  // or fail fast with a clear message about the JVM's security providers\n}","typeGuard":null,"tryCatchPattern":"try {\n  Job.getInstance(conf).submit();\n} catch (IOException io) {\n  if (io.getCause() instanceof java.security.NoSuchAlgorithmException) {\n    // JVM lacks HmacSHA1: disable local spill encryption or fix providers\n    conf.setBoolean(\"mapreduce.job.encrypted-intermediate-data\", false);\n  } else {\n    throw io;\n  }\n}","preventionTips":["Keep mapreduce.job.encrypted-intermediate-data=false in local and unit-test configs","Use a standard JDK (OpenJDK/Oracle 8+) for local runs; audit custom java.security files","Probe required algorithms in test setup when FIPS-restricted JVMs are in play"],"tags":["mapreduce","local-job-runner","encryption","jvm-security","hmac"],"backgroundTag":"unsupported-crypto-algorithm","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}