apache/beam · error · RuntimeException

Failed to retrieve or create secret bytes

Error message

Failed to retrieve or create secret bytes

What it means

GcpHsmGeneratedSecret.getSecretBytes either finds/creates a Secret Manager secret backed by Cloud KMS and reads its bytes. Any IOException or GeneralSecurityException during secret creation, addSecretVersion, or accessSecretVersion is wrapped into a RuntimeException('Failed to retrieve or create secret bytes', cause). The root cause is always attached.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcpHsmGeneratedSecret.java:160

      try {
        // Always retrieve remote secret as source-of-truth in case another thread created it
        AccessSecretVersionResponse response = client.accessSecretVersion(secretVersionName);
        return response.getPayload().getData().toByteArray();
      } catch (NotFoundException e) {
        LOG.info(
            "Secret version {} not found after re-check. Creating new secret and version.",
            secretVersionName.toString());
      }

      SecretPayload payload =
          SecretPayload.newBuilder().setData(ByteString.copyFrom(newKey)).build();
      client.addSecretVersion(secretName, payload);
      AccessSecretVersionResponse response = client.accessSecretVersion(secretVersionName);
      return response.getPayload().getData().toByteArray();

    } catch (IOException | GeneralSecurityException e) {
      throw new RuntimeException("Failed to retrieve or create secret bytes", e);
    }
  }

  private byte[] generateDek() throws IOException, GeneralSecurityException {
    int dekSize = 32;
    try (KeyManagementServiceClient client = KeyManagementServiceClient.create()) {
      // 1. Generate nonce_one. This doesn't need to have baked in randomness since the
      // actual randomness comes from KMS.
      byte[] nonceOne = new byte[dekSize];
      random.nextBytes(nonceOne);

      // 2. Encrypt to get nonce_two
      CryptoKeyName keyName = CryptoKeyName.of(projectId, locationId, keyRingId, keyId);
      EncryptResponse response = client.encrypt(keyName, ByteString.copyFrom(nonceOne));
      byte[] nonceTwo = response.getCiphertext().toByteArray();

      // 3. Generate DK
      byte[] dk = new byte[dekSize];

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the chained cause exception for the real API error
2. Enable the Secret Manager API: gcloud services enable secretmanager.googleapis.com
3. Grant the service account roles/secretmanager.admin and roles/cloudkms.cryptoKeyEncrypterDecrypter
4. Verify location_id matches the KMS key ring's region and the key exists

try { byte[] dek = hsmSecret.getSecretBytes(); }
catch (RuntimeException e) {
  Throwable cause = e.getCause(); // IOException | GeneralSecurityException
  LOG.error("Secret retrieval failed: " + cause, cause);
  throw new IllegalStateException("Check Secret Manager API, KMS permissions, and credentials", e);
}

When it happens

Trigger: KMS key not found or caller lacks cryptoKeyEncrypterDecrypter permission; Secret Manager API not enabled; secret creation/access API call failing (network, quota); invalid service credentials.

Common situations: Missing roles/cloudkms.cryptoKeyEncrypterDecrypter on the service account; secret in a region different from the KMS key ring; google-cloud-secretmanager not enabled on the project.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Invalid secret parameter %s
• Failed to retrieve secret bytes
• Invalid secret parameter %s
• Secret name must be specified in secret spec.
• Provided Secret must be in the form projects/{project}/secre

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/948a9fa5971d0894. Report an issue: GitHub.