apache/beam · error · ValueError

Missing required parameter(s) for GcpHsmGeneratedSecret

Error message

Missing required parameter(s) for GcpHsmGeneratedSecret: {sorted(list(missing))}

What it means

GcpHsmGeneratedSecret.from_dict requires all of 'project_id', 'location_id', 'key_ring_id', 'key_id', 'job_name' in the spec dict. If any are missing, it raises this ValueError listing the missing keys.

Solutions

Provide all five keys: project_id, location_id, key_ring_id, key_id, job_name.
Set job_name to your Beam job's name (it namespaces the derived secret).
Validate the dict against the required key set before calling from_dict.

Example fix

// before
GcpHsmGeneratedSecret.from_dict({'project_id': 'p', 'location_id': 'us', 'key_ring_id': 'r', 'key_id': 'k'})
// after
GcpHsmGeneratedSecret.from_dict({'project_id': 'p', 'location_id': 'us', 'key_ring_id': 'r', 'key_id': 'k', 'job_name': 'my-job'})

Defensive patterns

Strategy: validation

Validate before calling

REQUIRED = {'project_id', 'location_id', 'key_ring_id', 'key_id', 'job_name'}
missing = REQUIRED - set(spec)
if missing:
    raise ValueError(f'HSM spec missing: {sorted(missing)}')

Type guard

def is_complete_hsm_spec(d) -> bool:
    return isinstance(d, dict) and {'project_id', 'location_id', 'key_ring_id', 'key_id', 'job_name'} <= set(d)

Try / catch

try:
    secret = GcpHsmGeneratedSecret.from_dict(spec)
except ValueError as e:
    raise ConfigError(f'incomplete HSM secret spec: {e}') from e

Prevention

Keep the HSM spec as a dataclass with all five required fields
Include job_name whenever constructing the spec
Validate specs at pipeline startup, not at secret-use time

When it happens

Trigger: Calling GcpHsmGeneratedSecret.from_dict({'project_id': 'p', 'location_id': 'us', 'key_ring_id': 'ring'}) — omitting 'key_id' and/or 'job_name'; passing an empty dict.

Common situations: Partially filling the HSM spec because 'job_name' is easy to overlook; converting a plain GcpSecret spec to HSM without adding the KMS location/key-ring/key fields.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6b33b541ce1b38f0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/utils/secret.py:319

  def __eq__(self, other: Any) -> bool:
    if not isinstance(other, GcpHsmGeneratedSecret):
      return False
    return (
        self._project_id == other._project_id and
        self._location_id == other._location_id and
        self._key_ring_id == other._key_ring_id and
        self._key_id == other._key_id and
        getattr(self, '_job_name', None) == getattr(other, '_job_name', None))

  @classmethod
  def from_dict(cls, spec_dict: Dict[str, str]) -> 'GcpHsmGeneratedSecret':
    """Initialize GcpHsmGeneratedSecret from a dictionary specification."""
    allowed_keys = {
        'project_id', 'location_id', 'key_ring_id', 'key_id', 'job_name'
    }
    missing = allowed_keys - set(spec_dict.keys())
    if missing:
      raise ValueError(
          f"Missing required parameter(s) for GcpHsmGeneratedSecret: {sorted(list(missing))}"
      )
    invalid_keys = set(spec_dict.keys()) - allowed_keys
    if invalid_keys:
      raise ValueError(
          f"Invalid secret parameter {', '.join(sorted(invalid_keys))}")
    return cls(
        project_id=spec_dict['project_id'],
        location_id=spec_dict['location_id'],
        key_ring_id=spec_dict['key_ring_id'],
        key_id=spec_dict['key_id'],
        job_name=spec_dict['job_name'],
    )

  def get_secret_bytes(self) -> bytes:
    """Retrieves the secret bytes.

    If the secret version already exists in Secret Manager, it is retrieved.

View on GitHub (pinned to 12126d8942)