apache/beam · error · RuntimeException

Failed to retrieve or create secret bytes

Error message

Failed to retrieve or create secret bytes

What it means

GcpHsmGeneratedSecret.getSecretBytes either finds/creates a Secret Manager secret backed by Cloud KMS and reads its bytes. Any IOException or GeneralSecurityException during secret creation, addSecretVersion, or accessSecretVersion is wrapped into a RuntimeException('Failed to retrieve or create secret bytes', cause). The root cause is always attached.

Solutions

1. Read the chained cause exception for the real API error
2. Enable the Secret Manager API: gcloud services enable secretmanager.googleapis.com
3. Grant the service account roles/secretmanager.admin and roles/cloudkms.cryptoKeyEncrypterDecrypter
4. Verify location_id matches the KMS key ring's region and the key exists

try { byte[] dek = hsmSecret.getSecretBytes(); }
catch (RuntimeException e) {
  Throwable cause = e.getCause(); // IOException | GeneralSecurityException
  LOG.error("Secret retrieval failed: " + cause, cause);
  throw new IllegalStateException("Check Secret Manager API, KMS permissions, and credentials", e);
}

When it happens

Trigger: KMS key not found or caller lacks cryptoKeyEncrypterDecrypter permission; Secret Manager API not enabled; secret creation/access API call failing (network, quota); invalid service credentials.

Common situations: Missing roles/cloudkms.cryptoKeyEncrypterDecrypter on the service account; secret in a region different from the KMS key ring; google-cloud-secretmanager not enabled on the project.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Failed to retrieve secret bytes
• Invalid secret parameter
• Invalid secret parameter
• Provided Secret must be in the form projects/
• Secret name must be specified in secret spec.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/948a9fa5971d0894. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcpHsmGeneratedSecret.java:160

      try {
        // Always retrieve remote secret as source-of-truth in case another thread created it
        AccessSecretVersionResponse response = client.accessSecretVersion(secretVersionName);
        return response.getPayload().getData().toByteArray();
      } catch (NotFoundException e) {
        LOG.info(
            "Secret version {} not found after re-check. Creating new secret and version.",
            secretVersionName.toString());
      }

      SecretPayload payload =
          SecretPayload.newBuilder().setData(ByteString.copyFrom(newKey)).build();
      client.addSecretVersion(secretName, payload);
      AccessSecretVersionResponse response = client.accessSecretVersion(secretVersionName);
      return response.getPayload().getData().toByteArray();

    } catch (IOException | GeneralSecurityException e) {
      throw new RuntimeException("Failed to retrieve or create secret bytes", e);
    }
  }

  private byte[] generateDek() throws IOException, GeneralSecurityException {
    int dekSize = 32;
    try (KeyManagementServiceClient client = KeyManagementServiceClient.create()) {
      // 1. Generate nonce_one. This doesn't need to have baked in randomness since the
      // actual randomness comes from KMS.
      byte[] nonceOne = new byte[dekSize];
      random.nextBytes(nonceOne);

      // 2. Encrypt to get nonce_two
      CryptoKeyName keyName = CryptoKeyName.of(projectId, locationId, keyRingId, keyId);
      EncryptResponse response = client.encrypt(keyName, ByteString.copyFrom(nonceOne));
      byte[] nonceTwo = response.getCiphertext().toByteArray();

      // 3. Generate DK
      byte[] dk = new byte[dekSize];

View on GitHub (pinned to 12126d8942)