apache/beam · error · ValueError

Missing {required} in provider at line {SafeLineLoader.get_l

Error message

Missing {required} in provider at line {SafeLineLoader.get_line(spec)}

What it means

ExternalProvider.provider_from_spec (yaml_provider.py:266) validates that every provider spec has both 'type' and 'transforms' keys before dispatching to a registered provider constructor. If either is missing it raises a ValueError naming the offending key and the YAML line.

Source

Thrown at sdks/python/apache_beam/yaml/yaml_provider.py:266

          self._service,
          rearrange_based_on_discovery=True,
          managed_replacement=managed_replacement,
          **args)
    else:
      return type >> self.create_external_transform(urn, args)

  def create_external_transform(self, urn, args):
    return external.ExternalTransform(
        urn,
        external.ImplicitSchemaPayloadBuilder(args).payload(),
        self._service)

  @classmethod
  def provider_from_spec(cls, source_path, spec):
    from apache_beam.yaml.yaml_transform import SafeLineLoader
    for required in ('type', 'transforms'):
      if required not in spec:
        raise ValueError(
            f'Missing {required} in provider '
            f'at line {SafeLineLoader.get_line(spec)}')
    urns = SafeLineLoader.strip_metadata(spec['transforms'])
    type = spec['type']
    config = SafeLineLoader.strip_metadata(spec.get('config', {}))
    extra_params = set(SafeLineLoader.strip_metadata(spec).keys()) - {
        'transforms', 'type', 'config'
    }
    if extra_params:
      raise ValueError(
          f'Unexpected parameters in provider of type {type} '
          f'at line {SafeLineLoader.get_line(spec)}: {extra_params}')
    if config.get('version', None) == 'BEAM_VERSION':
      config['version'] = beam_version
    if type in cls._provider_types:
      try:
        constructor = cls._provider_types[type]
        if 'provider_base_path' in inspect.signature(constructor).parameters:

View on GitHub (pinned to 12126d8942)

Solutions

1. Add the missing key ('type' or 'transforms') to the provider spec at the reported line.
2. Check indentation so the keys are siblings at the provider level, not nested under config.
3. Validate the spec with a YAML linter / schema check before running the pipeline.

Example fix

# before
providers:
  - transforms:
      MyTransform: ...
# after
providers:
  - type: python
    transforms:
      MyTransform: ...

def validate_provider_spec(spec):
    missing = [k for k in ('type', 'transforms') if k not in spec]
    if missing:
        raise SystemExit(f'provider spec missing keys: {missing}')

def is_valid_provider_spec(spec) -> bool:
    return isinstance(spec, dict) and 'type' in spec and 'transforms' in spec

try:
    provider = ExternalProvider.provider_from_spec(src, spec)
except ValueError as e:
    raise SpecError(f'Bad provider spec: {e}') from e

When it happens

Trigger: A provider block in a YAML pipeline (or included spec) like {config: ...} without 'type', or a typed provider without the 'transforms' mapping, parsed via provider_from_spec.

Common situations: Hand-written YAML provider sections with indentation mistakes that drop a key; copying a provider template and deleting the transforms section; YAML anchors merging away 'type'.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Unexpected parameters in provider of type {type} at line {Sa
• Node ID cannot be empty
• Edge source and target cannot be empty
• Incompatible types: {weak_schema['type']} vs {strong_schema[
• Unknown output name "{tag}" from {by}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7d327c8291be2536. Report an issue: GitHub.