{"record":{"id":"d2b663a0b70b1391","repo":"apache/hadoop","slug":"too-many-retries-because-of-encryption-zone-operat","errorCode":null,"errorMessage":"Too many retries because of encryption zone operations","messagePattern":"Too many retries because of encryption zone operations","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java","lineNumber":318,"sourceCode":"          IOException e = re.unwrapRemoteException(\n              AccessControlException.class,\n              DSQuotaExceededException.class,\n              QuotaByStorageTypeExceededException.class,\n              FileAlreadyExistsException.class,\n              FileNotFoundException.class,\n              ParentNotDirectoryException.class,\n              NSQuotaExceededException.class,\n              RetryStartFileException.class,\n              SafeModeException.class,\n              UnresolvedPathException.class,\n              SnapshotAccessControlException.class,\n              UnknownCryptoProtocolVersionException.class);\n          if (e instanceof RetryStartFileException) {\n            if (retryCount > 0) {\n              shouldRetry = true;\n              retryCount--;\n            } else {\n              throw new IOException(\"Too many retries because of encryption\" +\n                  \" zone operations\", e);\n            }\n          } else {\n            throw e;\n          }\n        }\n      }\n      Preconditions.checkNotNull(stat, \"HdfsFileStatus should not be null!\");\n      final DFSOutputStream out;\n      if(stat.getErasureCodingPolicy() != null) {\n        out = new DFSStripedOutputStream(dfsClient, src, stat,\n            flag, progress, checksum, favoredNodes);\n      } else {\n        out = new DFSOutputStream(dfsClient, src, stat,\n            flag, progress, checksum, favoredNodes, true);\n      }\n      out.start();\n      return out;","sourceCodeStart":300,"sourceCodeEnd":336,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L300-L336","documentation":"newStreamForCreate() retries the NameNode create RPC when the NN responds with RetryStartFileException - the NN's signal that it cannot yet start the file, typically because the encryption-zone key is not currently retrievable (KMS slow or unreachable from the NN). After CREATE_RETRY_COUNT (10) attempts the client gives up and wraps the last exception in IOException('Too many retries because of encryption zone operations'). The retry count is a compile-time constant, not user-configurable, and the loop retries immediately with no backoff, so 10 quick failures can elapse in milliseconds.","triggerScenarios":"Calling create() on a path inside an HDFS encryption zone while the NameNode cannot fetch the zone's EDEK - KMS down, KMS unreachable from the NN, NN-to-KMS Kerberos/SSL broken, or backing key store (HSM) latency exceeding the NN's internal patience, causing RetryStartFileException ten times in a row.","commonSituations":"'Too many retries because of encryption zone operations' storms during KMS restarts or failovers; Kerberos keytab for the KMS principal expired on the NN; KMS backed by a slow HSM or its own backend keystore down; first writes to a newly created zone whose key material is still propagating.","solutions":["Restore KMS availability and the NN's ability to reach it: check the KMS process/HA pair, hadoop.security.keystore... key provider URI, NN-to-KMS Kerberos credentials, and SSL config; verify with 'hadoop key list -metadata' run as the NN's user.","Add an application-level retry with backoff around create() for files in encryption zones - the built-in 10 immediate retries are far too few to ride out a KMS restart.","Verify the zone/key state: 'hdfs crypto -listZones' and confirm the zone's key exists and is not corrupted in the KMS.","If using an HSM-backed provider, size its timeouts so the NN does not classify normal latency as retry-worthy."],"exampleFix":"// before\nFSDataOutputStream out = fs.create(path, true); // throws when KMS is briefly down\n\n// after - application retry for encryption-zone creates\nfor (int attempt = 1; ; attempt++) {\n  try {\n    FSDataOutputStream out = fs.create(path, true);\n    break;\n  } catch (IOException e) {\n    if (attempt >= 5 || !String.valueOf(e.getMessage()).contains(\"encryption zone\")) throw e;\n    Thread.sleep(1_000L * attempt); // outlast a KMS blip/restart\n  }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"IOException last = null;\nfor (int attempt = 1; attempt <= 5; attempt++) {\n  try {\n    return fs.create(path, true);\n  } catch (IOException e) {\n    if (!String.valueOf(e.getMessage()).contains(\"encryption zone\")) throw e; // not EZ-related\n    last = e;\n    Thread.sleep(1_000L * attempt); // outlast KMS restart/failover (built-in retries are immediate)\n  }\n}\nthrow last;","preventionTips":["Monitor KMS health and keep an HA pair - this error is almost always a KMS availability problem.","Keep NN-to-KMS Kerberos keytabs and SSL certs valid; expiry produces the same retry storm.","Wrap creates of files inside encryption zones in an application-level backoff retry (the 10 internal retries have no sleep).","Alert on repeated RetryStartFileException in NN logs to catch KMS degradation before writers fail."],"tags":["hdfs","hdfs-client","encryption-zone","kms","create","retry"],"backgroundTag":"kms-unavailable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}