apache/beam · error · RuntimeError

Failed to retrieve or create secret bytes for secret

Error message

Failed to retrieve or create secret bytes for secret {self._secret_version_name} with exception {e}

What it means

GcpHsmGeneratedSecret.get_secret_bytes either retrieves an existing secret version or creates one via the Secret Manager / KMS APIs. Any exception during that retrieve-or-create flow (API errors, permission denied, KMS key not found, bad crypto config, network) is re-raised as a RuntimeError naming the secret version and the original error.

Solutions

Inspect the wrapped exception and fix the root cause (NotFound → verify project/location/keyring/key; PermissionDenied → add required IAM roles).
Verify the KMS key exists in the specified location_id and key_ring_id.
Grant the runtime service account roles/secretmanager.admin and roles/cloudkms.cryptoKeyEncrypterDecrypter.
Enable Secret Manager and Cloud KMS APIs on the project.
Retry with backoff on transient gRPC/network errors.

Example fix

// before
bytes = hsm_secret.get_secret_bytes()  # PermissionDenied
// after
# grant roles: secretmanager.admin, cloudkms.cryptoKeyEncrypterDecrypter
# verify key ring/key exist in location_id
bytes = hsm_secret.get_secret_bytes()

Defensive patterns

Strategy: try-catch

Validate before calling

from google.cloud import kms_v1, secretmanager_v1
kms = kms_v1.KeyManagementServiceClient()
kms_key = kms.get_crypto_key(request={'name': f'projects/{p}/locations/{l}/keyRings/{r}/cryptoKeys/{k}'})  # verify KMS key exists

Type guard

def hsm_resources_exist(project, location, key_ring, key_id) -> bool:
    from google.cloud import kms_v1
    kms = kms_v1.KeyManagementServiceClient()
    try:
        kms.get_crypto_key(request={'name': f'projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{key_id}'})
        return True
    except Exception:
        return False

Try / catch

try:
    data = hsm_secret.get_secret_bytes()
except RuntimeError as e:
    logging.error('HSM secret flow failed: %s', e)
    if 'PermissionDenied' in str(e): fix_iam_roles()
    elif 'NotFound' in str(e): verify_kms_and_secret()
    else: raise

Prevention

Create the key ring/key in the same location_id as the spec
Grant secretmanager.admin and cloudkms.cryptoKeyEncrypterDecrypter to the runtime SA
Enable Secret Manager and Cloud KMS APIs
Use a stable job_name so secret lookup/reuse is consistent
Add retry with backoff for transient gRPC errors

When it happens

Trigger: Calling get_secret_bytes() when the Cloud KMS key/keyring doesn't exist, the service account lacks secretmanager.admin + cloudkms.cryptoKeyEncrypterDecrypter, the secret already exists but is corrupted/inaccessible, or the API calls fail transiently.

Common situations: KMS key ring created in the wrong location; job_name mismatch causing lookup of a secret that doesn't exist; IAM roles missing for the create path; Secret Manager API not enabled.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5aab50354f5eda85. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/utils/secret.py:403

            request={"name": secret_version_path})
        return response.payload.data
      except api_exceptions.NotFound:
        _LOGGER.info(
            "Secret version %s not found. "
            "Creating new secret and version.",
            secret_version_path)
      client.add_secret_version(
          request={
              "parent": secret_path, "payload": {
                  "data": new_key
              }
          })
      response = client.access_secret_version(
          request={"name": secret_version_path})
      return response.payload.data

    except Exception as e:
      raise RuntimeError(
          f'Failed to retrieve or create secret bytes for secret '
          f'{self._secret_version_name} with exception {e}')

  def generate_dek(self, dek_size: int = 32) -> bytes:
    """Generates a new Data Encryption Key (DEK) using an HSM-backed key.

    This function follows a key derivation process that incorporates entropy
    from the HSM-backed key into the nonce used for key derivation.

    Args:
      dek_size: The size of the DEK to generate.

    Returns:
        A new DEK of the specified size, url-safe base64-encoded.
    """
    try:
      import base64
      import os

View on GitHub (pinned to 12126d8942)