{"record":{"id":"5aab50354f5eda85","repo":"apache/beam","slug":"failed-to-retrieve-or-create-secret-bytes-for-secret-self","errorCode":null,"errorMessage":"Failed to retrieve or create secret bytes for secret {self._secret_version_name} with exception {e}","messagePattern":"Failed to retrieve or create secret bytes for secret (.+?) with exception (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/utils/secret.py","lineNumber":403,"sourceCode":"            request={\"name\": secret_version_path})\n        return response.payload.data\n      except api_exceptions.NotFound:\n        _LOGGER.info(\n            \"Secret version %s not found. \"\n            \"Creating new secret and version.\",\n            secret_version_path)\n      client.add_secret_version(\n          request={\n              \"parent\": secret_path, \"payload\": {\n                  \"data\": new_key\n              }\n          })\n      response = client.access_secret_version(\n          request={\"name\": secret_version_path})\n      return response.payload.data\n\n    except Exception as e:\n      raise RuntimeError(\n          f'Failed to retrieve or create secret bytes for secret '\n          f'{self._secret_version_name} with exception {e}')\n\n  def generate_dek(self, dek_size: int = 32) -> bytes:\n    \"\"\"Generates a new Data Encryption Key (DEK) using an HSM-backed key.\n\n    This function follows a key derivation process that incorporates entropy\n    from the HSM-backed key into the nonce used for key derivation.\n\n    Args:\n      dek_size: The size of the DEK to generate.\n\n    Returns:\n        A new DEK of the specified size, url-safe base64-encoded.\n    \"\"\"\n    try:\n      import base64\n      import os","sourceCodeStart":385,"sourceCodeEnd":421,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/utils/secret.py#L385-L421","documentation":"GcpHsmGeneratedSecret.get_secret_bytes either retrieves an existing secret version or creates one via the Secret Manager / KMS APIs. Any exception during that retrieve-or-create flow (API errors, permission denied, KMS key not found, bad crypto config, network) is re-raised as a RuntimeError naming the secret version and the original error.","triggerScenarios":"Calling get_secret_bytes() when the Cloud KMS key/keyring doesn't exist, the service account lacks secretmanager.admin + cloudkms.cryptoKeyEncrypterDecrypter, the secret already exists but is corrupted/inaccessible, or the API calls fail transiently.","commonSituations":"KMS key ring created in the wrong location; job_name mismatch causing lookup of a secret that doesn't exist; IAM roles missing for the create path; Secret Manager API not enabled.","solutions":["Inspect the wrapped exception and fix the root cause (NotFound → verify project/location/keyring/key; PermissionDenied → add required IAM roles).","Verify the KMS key exists in the specified location_id and key_ring_id.","Grant the runtime service account roles/secretmanager.admin and roles/cloudkms.cryptoKeyEncrypterDecrypter.","Enable Secret Manager and Cloud KMS APIs on the project.","Retry with backoff on transient gRPC/network errors."],"exampleFix":"// before\nbytes = hsm_secret.get_secret_bytes()  # PermissionDenied\n// after\n# grant roles: secretmanager.admin, cloudkms.cryptoKeyEncrypterDecrypter\n# verify key ring/key exist in location_id\nbytes = hsm_secret.get_secret_bytes()","handlingStrategy":"try-catch","validationCode":"from google.cloud import kms_v1, secretmanager_v1\nkms = kms_v1.KeyManagementServiceClient()\nkms_key = kms.get_crypto_key(request={'name': f'projects/{p}/locations/{l}/keyRings/{r}/cryptoKeys/{k}'})  # verify KMS key exists","typeGuard":"def hsm_resources_exist(project, location, key_ring, key_id) -> bool:\n    from google.cloud import kms_v1\n    kms = kms_v1.KeyManagementServiceClient()\n    try:\n        kms.get_crypto_key(request={'name': f'projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{key_id}'})\n        return True\n    except Exception:\n        return False","tryCatchPattern":"try:\n    data = hsm_secret.get_secret_bytes()\nexcept RuntimeError as e:\n    logging.error('HSM secret flow failed: %s', e)\n    if 'PermissionDenied' in str(e): fix_iam_roles()\n    elif 'NotFound' in str(e): verify_kms_and_secret()\n    else: raise\n","preventionTips":["Create the key ring/key in the same location_id as the spec","Grant secretmanager.admin and cloudkms.cryptoKeyEncrypterDecrypter to the runtime SA","Enable Secret Manager and Cloud KMS APIs","Use a stable job_name so secret lookup/reuse is consistent","Add retry with backoff for transient gRPC errors"],"tags":["python","apache-beam","hsm","kms","secret-manager","api-error"],"backgroundTag":"api-error-response","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}