apache/hadoop · error · IOException

Failed to compute unencrypted length

Error message

Failed to compute unencrypted length

What it means

IOException from CSEUtils when computing the plaintext length of a client-side-encrypted (CSE) object. For CSE, S3A stores a padding block at the tail, so for contentLength >= CSE_PADDING_LENGTH the code issues a ranged GET over [contentLength - padding, contentLength] and counts the returned bytes to recover the true unencrypted size. Any failure of that ranged GET (permission, throttling, network, deserialization of the encrypted tail) is caught and rethrown as this wrapping IOException with the original as cause.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CSEUtils.java:147

          && !plaintextLength.isEmpty()) {
        return Long.parseLong(plaintextLength);
      }
    }

    // identify the length by doing a ranged GET operation.
    if (contentLength >= CSE_PADDING_LENGTH) {
      long minPlaintextLength = contentLength - CSE_PADDING_LENGTH;
      if (minPlaintextLength < 0) {
        minPlaintextLength = 0;
      }
      try (InputStream is = store.getRangedS3Object(key, minPlaintextLength, contentLength)) {
        int i = 0;
        while (is.read() != -1) {
          i++;
        }
        return minPlaintextLength + i;
      } catch (Exception e) {
        throw new IOException("Failed to compute unencrypted length", e);
      }
    }
    return contentLength;
  }

  /**
   * Creates encryption materials for client-side encryption based on the specified algorithm.
   *
   * Supports two types of client-side encryption:
   * <ul>
   *   <li>CSE_KMS: Uses AWS KMS for key management</li>
   *   <li>CSE_CUSTOM: Uses a custom cryptographic implementation</li>
   * </ul>
   *
   * @param conf      The configuration containing encryption settings
   * @param bucket    The S3 bucket name for which encryption materials are being created
   * @param algorithm The encryption algorithm to use (CSE_KMS or CSE_CUSTOM)
   * @return         CSEMaterials configured with the appropriate encryption settings

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the wrapped cause (e.getCause()) -- AccessDenied points to KMS/key-policy issues, NoSuchKey to a deleted object, 503/timeout to transient S3 conditions
2. Verify the reading principal can decrypt with the key that encrypted the object (kms:Decrypt grant and key policy for cross-account)
3. Retry the operation for transient causes; the length computation is idempotent
4. If the object may be changing concurrently, quiesce writers before statting CSE objects

try {
  FileStatus st = fs.getFileStatus(csePath);
} catch (IOException e) {
  if ("Failed to compute unencrypted length".equals(e.getMessage())
      && e.getCause() instanceof AccessDeniedException /* or SDK AccessDenied */) {
    // key-material permission problem: not retryable, fix KMS/key policy
    throw new SecurityException("Cannot decrypt CSE object tail", e);
  }
  // otherwise transient: retry with backoff
}

When it happens

Trigger: Calling getFileStatus (or any API that needs the length) on an object written with fs.s3a.encryption.algorithm=CSE-KMS or CSE-CUSTOM when the tail ranged read fails: no s3:GetObject permission on the key material, KMS key disabled or inaccessible, object deleted mid-operation, or a transient S3 error.

Common situations: Reader lacks grants to the KMS key used to encrypt the data; cross-account access without key policy updates; throttled S3 responses during metadata-heavy workloads; object lifecycle deleted the object between listing and stat.

Related errors

• Can't cast key for ${name} in keystore ${path} to a KeyMetad
• Can't get metadata for ${name} from keystore ${path}
• Can't recover key for ${name} from keystore ${path}
• Multi-part uploader not supported for Client side encryption
• Invalid client side encryption algorithm. Only CSE-KMS and C

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6e719127a0b8cac8. Report an issue: GitHub.