apache/beam · warning

The 'spec' parameter appears to be a JSON specification…

Error message

The 'spec' parameter appears to be a JSON specification, but 'secret_manager' is not set. Defaulting to Raw.

What it means

Secret.from_json (via parse_secret_option) parses secret options. If a 'spec' parameter looks like a JSON specification dict but no 'secret_manager' was set, the code cannot build a managed secret and falls back to RawSecret, logging and warning that it 'Defaulting to Raw'.

Solutions

1. Set the secret_manager option (e.g. gcpmanager/awsmanager/vaultmanager) alongside spec so the correct managed Secret is constructed.
2. Use the documented secret option format: --secret=<manager>:<project>:<secret_name>:<version>.
3. Verify the option string is not accidentally interpreted as a JSON spec (quote/escape correctly on the CLI).
4. If raw is actually intended, suppress the warning or pass the secret directly without the JSON-spec shape.

Example fix

// before
--secret='{"secret_id": "my-secret"}'
// after
--secret=gcpmanager:my-project:my-secret:latest

spec = pipeline_options.get_all_options().get('secret')
if spec and spec.lstrip().startswith('{') and 'secret_manager' not in spec:
    raise ValueError('secret spec looks like JSON but secret_manager is missing')

import warnings
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter('always')
    secret = Secret.from_json(spec)
    if any('Defaulting to Raw' in str(x.message) for x in w):
        raise ValueError('secret fell back to Raw; set secret_manager')

When it happens

Trigger: Passing a JSON-shaped 'spec' secret option (e.g. --secret=spec='{...}') without also setting secret_manager, e.g. a malformed/mistyped option string where the manager key was omitted or misspelled.

Common situations: Users intending to use GCP Secret Manager / AWS / HashiVault secrets but forgetting the secret_manager option, or hand-writing the secret option string instead of the documented format, so the raw secret string gets embedded in the pipeline instead of a manager-backed reference.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Failed to get secret.
• Failed to parse secret option
• Invalid secret type , currently supported types
• Secret option string cannot be null
• Secret string must contain a valid type parameter

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/867f8da585e3250a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/utils/secret.py:173

          secret_cls = secret_cls_entry
        if isinstance(spec_dict, dict) and hasattr(secret_cls, 'from_dict'):
          return secret_cls.from_dict(spec_dict)
        elif isinstance(spec_dict, dict):
          return secret_cls(**spec_dict)
        else:
          return secret_cls(spec)
      else:
        raise ValueError(
            f"Unsupported secret manager: '{secret_manager_name}'. Currently supported options: 'GoogleCloudSecretManager', 'GoogleCloudHsmGeneratedSecretManager'."
        )

    # If secret_manager is not set or empty, check if spec is a JSON specification dict
    if spec_dict is not None:
      msg = (
          "The 'spec' parameter appears to be a JSON specification, but "
          "'secret_manager' is not set. Defaulting to Raw.")
      _LOGGER.warning(msg)
      warnings.warn(msg, UserWarning)

    return RawSecret(spec)


class RawSecret(Secret):
  """Secret implementation wrapping a raw secret string or bytes directly."""
  def __init__(self, secret: Union[str, bytes]):
    super().__init__()
    if isinstance(secret, str):
      self._secret = secret.encode("utf-8")
    else:
      self._secret = secret

  def get_secret_bytes(self) -> bytes:
    return self._secret

  def __eq__(self, other: Any) -> bool:
    if not isinstance(other, RawSecret):

View on GitHub (pinned to 12126d8942)