{"record":{"id":"778047db3f91cea4","repo":"apache/beam","slug":"failed-to-generate-dek-with-exception-e","errorCode":null,"errorMessage":"Failed to generate DEK with exception {e}","messagePattern":"Failed to generate DEK with exception (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/utils/secret.py","lineNumber":453,"sourceCode":"          request={\n              'name': key_path, 'plaintext': nonce_one\n          })\n      nonce_two = response.ciphertext\n\n      # 3. Generate a Derivation Key (DK)\n      dk = os.urandom(dek_size)\n\n      # 4. Use a KDF to derive the DEK using DK and nonce_two\n      hkdf = HKDF(\n          algorithm=hashes.SHA256(),\n          length=dek_size,\n          salt=nonce_two,\n          info=None,\n      )\n      dek = hkdf.derive(dk)\n      return base64.urlsafe_b64encode(dek)\n    except Exception as e:\n      raise RuntimeError(f'Failed to generate DEK with exception {e}')\n\n\n_SECRET_TYPE_TO_SECRET_MANAGER: Dict[str, str] = {\n    \"gcpsecret\": \"GoogleCloudSecretManager\",\n    \"gcphsmgeneratedsecret\": \"GoogleCloudHsmGeneratedSecretManager\",\n}\n\n_SECRET_CLASSES: Dict[str, Any] = {\n    \"googlecloudsecretmanager\": \"GcpSecret\",\n    \"googlecloudhsmgeneratedsecretmanager\": \"GcpHsmGeneratedSecret\",\n}","sourceCodeStart":435,"sourceCodeEnd":464,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/utils/secret.py#L435-L464","documentation":"generate_dek derives a Data Encryption Key via HKDF from key material obtained through the Cloud KMS HSM key. Any failure in that derivation chain (failed KMS encrypt/decrypt, HKDF error, base64 issue) is re-raised as a RuntimeError with the underlying exception.","triggerScenarios":"generate_dek() fails because get_secret_bytes upstream failed (KMS/auth errors), dek_size is invalid for HKDF, or the derived-key crypto operation raised (e.g. cryptography library error).","commonSituations":"Called indirectly from get_secret_bytes when the HSM key is unavailable or IAM is misconfigured; oversized dek_size causing HKDF length limits to be exceeded.","solutions":["Fix the wrapped root exception — most often an upstream KMS/Secret Manager failure (see get_secret_bytes handling).","Use a normal dek_size (e.g. 32) within HKDF output limits.","Ensure the cryptography/HKDF dependencies are installed and functional.","Retry on transient KMS errors with backoff."],"exampleFix":"// before\ndek = secret.generate_dek(dek_size=999999)  # HKDF limit exceeded\n// after\ndek = secret.generate_dek(dek_size=32)","handlingStrategy":"try-catch","validationCode":"key_material = hsm_secret.get_secret_bytes()  # resolve upstream failures first\ndek = hsm_secret.generate_dek(dek_size=32)  # keep dek_size within HKDF limits","typeGuard":"def can_generate_dek(secret, dek_size: int) -> bool:\n    return 16 <= dek_size <= 1024","tryCatchPattern":"try:\n    dek = hsm_secret.generate_dek(dek_size)\nexcept RuntimeError as e:\n    logging.error('DEK generation failed: %s', e)\n    raise EncryptionError('cannot derive DEK; check KMS access and dek_size') from e\n","preventionTips":["Use dek_size=32 (or another standard size)","Ensure get_secret_bytes works before generate_dek","Keep cryptography library up to date","Retry on transient KMS errors"],"tags":["python","apache-beam","hsm","kms","encryption","hkdf"],"backgroundTag":"api-request-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}