What causes "Failed to generate DEK with exception {e}" in apache/beam?

Fix the wrapped root exception — most often an upstream KMS/Secret Manager failure (see get_secret_bytes handling).

apache/beam · error · RuntimeError

Failed to generate DEK with exception

Error message

Failed to generate DEK with exception {e}

What it means

generate_dek derives a Data Encryption Key via HKDF from key material obtained through the Cloud KMS HSM key. Any failure in that derivation chain (failed KMS encrypt/decrypt, HKDF error, base64 issue) is re-raised as a RuntimeError with the underlying exception.

Solutions

Fix the wrapped root exception — most often an upstream KMS/Secret Manager failure (see get_secret_bytes handling).
Use a normal dek_size (e.g. 32) within HKDF output limits.
Ensure the cryptography/HKDF dependencies are installed and functional.
Retry on transient KMS errors with backoff.

Example fix

// before
dek = secret.generate_dek(dek_size=999999)  # HKDF limit exceeded
// after
dek = secret.generate_dek(dek_size=32)

Defensive patterns

Strategy: try-catch

Validate before calling

key_material = hsm_secret.get_secret_bytes()  # resolve upstream failures first
dek = hsm_secret.generate_dek(dek_size=32)  # keep dek_size within HKDF limits

Type guard

def can_generate_dek(secret, dek_size: int) -> bool:
    return 16 <= dek_size <= 1024

Try / catch

try:
    dek = hsm_secret.generate_dek(dek_size)
except RuntimeError as e:
    logging.error('DEK generation failed: %s', e)
    raise EncryptionError('cannot derive DEK; check KMS access and dek_size') from e

Prevention

Use dek_size=32 (or another standard size)
Ensure get_secret_bytes works before generate_dek
Keep cryptography library up to date
Retry on transient KMS errors

When it happens

Trigger: generate_dek() fails because get_secret_bytes upstream failed (KMS/auth errors), dek_size is invalid for HKDF, or the derived-key crypto operation raised (e.g. cryptography library error).

Common situations: Called indirectly from get_secret_bytes when the HSM key is unavailable or IAM is misconfigured; oversized dek_size causing HKDF length limits to be exceeded.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/778047db3f91cea4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/utils/secret.py:453

          request={
              'name': key_path, 'plaintext': nonce_one
          })
      nonce_two = response.ciphertext

      # 3. Generate a Derivation Key (DK)
      dk = os.urandom(dek_size)

      # 4. Use a KDF to derive the DEK using DK and nonce_two
      hkdf = HKDF(
          algorithm=hashes.SHA256(),
          length=dek_size,
          salt=nonce_two,
          info=None,
      )
      dek = hkdf.derive(dk)
      return base64.urlsafe_b64encode(dek)
    except Exception as e:
      raise RuntimeError(f'Failed to generate DEK with exception {e}')


_SECRET_TYPE_TO_SECRET_MANAGER: Dict[str, str] = {
    "gcpsecret": "GoogleCloudSecretManager",
    "gcphsmgeneratedsecret": "GoogleCloudHsmGeneratedSecretManager",
}

_SECRET_CLASSES: Dict[str, Any] = {
    "googlecloudsecretmanager": "GcpSecret",
    "googlecloudhsmgeneratedsecretmanager": "GcpHsmGeneratedSecret",
}

View on GitHub (pinned to 12126d8942)