apache/beam · error · ValueError

Missing config in ML transform spec {spec}

Error message

Missing config in ML transform spec {spec}

What it means

_config_to_obj found a 'type' in the ML transform spec but no 'config' mapping; providers are constructed from their configuration dict, so its absence leaves nothing to instantiate the transform with.

Source

Thrown at sdks/python/apache_beam/yaml/yaml_ml.py:593

      pcoll | RunInference(
          model_handler=KeyedModelHandler(
              model_handler_provider.underlying_handler()).with_preprocess_fn(
                  model_handler_provider._preprocess_fn_internal()).
          with_postprocess_fn(
              model_handler_provider._postprocess_fn_internal()),
          inference_args=inference_args)
      | beam.Map(
          lambda row: beam.Row(
              **{
                  **row[0]._asdict(), str(inference_tag): row[1]
              })).with_output_types(schema))


def _config_to_obj(spec):
  if 'type' not in spec:
    raise ValueError(f"Missing type in ML transform spec {spec}")
  if 'config' not in spec:
    raise ValueError(f"Missing config in ML transform spec {spec}")
  constructor = _transform_constructors.get(spec['type'])
  if constructor is None:
    raise ValueError("Unknown ML transform type: %r" % spec['type'])
  return constructor(**spec['config'])


@beam.ptransform.ptransform_fn
def ml_transform(
    pcoll,
    write_artifact_location: Optional[str] = None,
    read_artifact_location: Optional[str] = None,
    transforms: Optional[list[Any]] = None):
  if MLTransform is None:
    raise ValueError(
        'No MLTransform found. Please install tensorflow-transform or '
        'sentence-transformers to use this transform.')
  options.YamlOptions.check_enabled(pcoll.pipeline, 'ML')
  result_ml_transform = MLTransform(

View on GitHub (pinned to 12126d8942)

Solutions

1. Add a 'config' key with the transform options (use {} for defaults)
2. Fix YAML indentation so options are nested under config
3. Check key spelling: it must be exactly 'config'

Example fix

# before
- name: inference
  type: RunInference
# after
- name: inference
  type: RunInference
  config:
    model_handler:
      type: VertexAI
      config: {endpoint_id: '123', project: 'my-project'}

def check_spec_config(spec):
    if 'config' not in spec:
        raise ValueError(f'ML transform spec needs config: {spec}')

def has_config(spec):
    return isinstance(spec, dict) and isinstance(spec.get('config'), dict)

When it happens

Trigger: An ML transform spec like {'type': 'RunInference'} with no 'config' key; config misspelled or misplaced due to YAML indentation.

Common situations: Transforms whose options were accidentally placed as siblings of type/config; specs copied without their config block; empty-config transforms where users omit the key instead of passing config: {}.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Unexpected parameters in model_handler: {extra_params}
• Missing parameters in model_handler: {missing_params}
• Missing type in ML transform spec {spec}
• Node ID cannot be empty
• Edge source and target cannot be empty

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/37292b5fd9871616. Report an issue: GitHub.