apache/hadoop · error · IOException

Too many retries because of encryption zone operations

Error message

Too many retries because of encryption zone operations

What it means

newStreamForCreate() retries the NameNode create RPC when the NN responds with RetryStartFileException - the NN's signal that it cannot yet start the file, typically because the encryption-zone key is not currently retrievable (KMS slow or unreachable from the NN). After CREATE_RETRY_COUNT (10) attempts the client gives up and wraps the last exception in IOException('Too many retries because of encryption zone operations'). The retry count is a compile-time constant, not user-configurable, and the loop retries immediately with no backoff, so 10 quick failures can elapse in milliseconds.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java:318

          IOException e = re.unwrapRemoteException(
              AccessControlException.class,
              DSQuotaExceededException.class,
              QuotaByStorageTypeExceededException.class,
              FileAlreadyExistsException.class,
              FileNotFoundException.class,
              ParentNotDirectoryException.class,
              NSQuotaExceededException.class,
              RetryStartFileException.class,
              SafeModeException.class,
              UnresolvedPathException.class,
              SnapshotAccessControlException.class,
              UnknownCryptoProtocolVersionException.class);
          if (e instanceof RetryStartFileException) {
            if (retryCount > 0) {
              shouldRetry = true;
              retryCount--;
            } else {
              throw new IOException("Too many retries because of encryption" +
                  " zone operations", e);
            }
          } else {
            throw e;
          }
        }
      }
      Preconditions.checkNotNull(stat, "HdfsFileStatus should not be null!");
      final DFSOutputStream out;
      if(stat.getErasureCodingPolicy() != null) {
        out = new DFSStripedOutputStream(dfsClient, src, stat,
            flag, progress, checksum, favoredNodes);
      } else {
        out = new DFSOutputStream(dfsClient, src, stat,
            flag, progress, checksum, favoredNodes, true);
      }
      out.start();
      return out;

View on GitHub (pinned to 2add963021)

Solutions

Restore KMS availability and the NN's ability to reach it: check the KMS process/HA pair, hadoop.security.keystore... key provider URI, NN-to-KMS Kerberos credentials, and SSL config; verify with 'hadoop key list -metadata' run as the NN's user.
Add an application-level retry with backoff around create() for files in encryption zones - the built-in 10 immediate retries are far too few to ride out a KMS restart.
Verify the zone/key state: 'hdfs crypto -listZones' and confirm the zone's key exists and is not corrupted in the KMS.
If using an HSM-backed provider, size its timeouts so the NN does not classify normal latency as retry-worthy.

Example fix

// before
FSDataOutputStream out = fs.create(path, true); // throws when KMS is briefly down

// after - application retry for encryption-zone creates
for (int attempt = 1; ; attempt++) {
  try {
    FSDataOutputStream out = fs.create(path, true);
    break;
  } catch (IOException e) {
    if (attempt >= 5 || !String.valueOf(e.getMessage()).contains("encryption zone")) throw e;
    Thread.sleep(1_000L * attempt); // outlast a KMS blip/restart
  }
}

Defensive patterns

Strategy: retry

Try / catch

IOException last = null;
for (int attempt = 1; attempt <= 5; attempt++) {
  try {
    return fs.create(path, true);
  } catch (IOException e) {
    if (!String.valueOf(e.getMessage()).contains("encryption zone")) throw e; // not EZ-related
    last = e;
    Thread.sleep(1_000L * attempt); // outlast KMS restart/failover (built-in retries are immediate)
  }
}
throw last;

Prevention

Monitor KMS health and keep an HA pair - this error is almost always a KMS availability problem.
Keep NN-to-KMS Kerberos keytabs and SSL certs valid; expiry produces the same retry storm.
Wrap creates of files inside encryption zones in an application-level backoff retry (the 10 internal retries have no sleep).
Alert on repeated RetryStartFileException in NN logs to catch KMS degradation before writers fail.

When it happens

Trigger: Calling create() on a path inside an HDFS encryption zone while the NameNode cannot fetch the zone's EDEK - KMS down, KMS unreachable from the NN, NN-to-KMS Kerberos/SSL broken, or backing key store (HSM) latency exceeding the NN's internal patience, causing RetryStartFileException ten times in a row.

Common situations: 'Too many retries because of encryption zone operations' storms during KMS restarts or failovers; Kerberos keytab for the KMS principal expired on the NN; KMS backed by a slow HSM or its own backend keystore down; first writes to a newly created zone whose key material is still propagating.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d2b663a0b70b1391. Report an issue: GitHub.