apache/beam · error · IllegalArgumentException

Failed to get secret.

Error message

Failed to get secret.

What it means

FileAwareFactoryFn resolves secret placeholders in Kafka config values via a secrets manager. If secret processing throws IllegalArgumentException (e.g. malformed secret id, secret lookup rejected), it is rethrown as 'Failed to get secret.' with the cause attached.

Solutions

1. Verify the secret exists and its id in the config matches the expected format
2. Grant the job's service account permission to read the secret
3. Inspect the wrapped IllegalArgumentException cause for the precise failure

Example fix

// before
"password": "{{secret:db-pass}" // malformed id
// after
"password": "{{secret:projects/p/secrets/db-pass}}" // correct format

if (secretId == null || secretId.isEmpty()) { throw new IllegalArgumentException("Empty secret id in config value"); } // plus verify the secret exists via the secrets manager API before submit

boolean looksLikeSecretRef(String v) { return v != null && v.startsWith(SECRET_VALUE_PREFIX) && v.length() > SECRET_VALUE_PREFIX.length(); }

try { apply(configValue); } catch (IllegalArgumentException e) { if ("Failed to get secret.".equals(e.getMessage())) { /* fix secret id or IAM permissions */ } else throw e; }

When it happens

Trigger: A config value contains a secret placeholder whose id is malformed or not resolvable at apply() time, causing getSecretWithCache/processSecret to throw IllegalArgumentException.

Common situations: Secret not created or deleted in the secrets manager, wrong secret id format in config, job lacks IAM permission to read the secret so the lookup fails.

Related errors

• consumerPollingTimeout should be > 0.
• Failed to parse secret option
• Failed trying to process value for key
• Invalid secret type , currently supported types
• Not yet implemented.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1ed9d4fba507e4b3. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFn.java:140

              if (externalPath != null) {
                try {
                  String tmpPath = replacePathWithLocal(externalPath);
                  String localPath = downloadExternalFile(externalPath, tmpPath);
                  matcher.appendReplacement(sb, Matcher.quoteReplacement(localPath));
                  LOG.info("Downloaded {} to {}", externalPath, localPath);
                } catch (IOException io) {
                  throw new IOException("Failed to download file : " + externalPath, io);
                }
              } else if (secretValue != null) {
                try {
                  String secretId = secretValue.substring(SECRET_VALUE_PREFIX.length());
                  String processedSecret =
                      processSecret(originalValue, secretId, getSecretWithCache(secretId));

                  matcher.appendReplacement(sb, Matcher.quoteReplacement(processedSecret));
                } catch (IllegalArgumentException ia) {
                  throw new IllegalArgumentException("Failed to get secret.", ia);
                }
              } else if (secretFile != null) {
                throw new UnsupportedOperationException("Not yet implemented.");
              }
            }
            matcher.appendTail(sb);
            String processedValue = sb.toString();
            processedConfig.put(key, processedValue);
          }
        } catch (IOException ex) {
          throw new RuntimeException("Failed trying to process value for key " + key + ".", ex);
        }
      }
    } catch (IOException e) {
      throw new RuntimeException("Failed trying to process extra files.", e);
    }

    return createObject(processedConfig);

View on GitHub (pinned to 12126d8942)