apache/beam · error · ValueError

Must specify one of 'callable' or 'path' and 'name' for

Error message

Must specify one of 'callable' or 'path' and 'name' for {typ} function.

What it means

`_parse_config` in yaml_ml.py:100 requires one of two ways to identify the processing function: an inline `callable`, or both a `path` (script file) and `name` (function in that script). If neither form is fully provided, ValueError is raised.

Solutions

Provide both `path` and `name` when loading from a script: {path: preprocess.py, name: my_fn}.
Or provide an inline `callable` instead of path/name.
Verify the YAML preprocess block is not empty and all required keys are present.

Example fix

# before
preprocess: {path: preprocess.py}
# after
preprocess: {path: preprocess.py, name: my_preprocess_fn}

Defensive patterns

Strategy: validation

Validate before calling

cfg = processing_transform if isinstance(processing_transform, dict) else {}
ok = ('callable' in cfg) or ('path' in cfg and 'name' in cfg)
assert ok, 'must provide callable, or both path and name'

Type guard

def has_complete_fn_config(cfg):
    return 'callable' in cfg or ('path' in cfg and 'name' in cfg)

Try / catch

try:
    fn = parse_processing_transform(spec, typ)
except ValueError as e:
    raise YamlConfigError('preprocess needs callable or path+name') from e

Prevention

Always specify function `name` alongside `path` for script-based configs.
Never leave a preprocess block empty; omit it only if the handler has a default.
Templatize preprocess configs to keep required keys present.

When it happens

Trigger: Passing an empty dict or a dict missing keys, e.g. {path: preprocess.py} without `name`, or {name: my_fn} without `path`, or no preprocess config at all where one is required.

Common situations: Forgetting the `name` of the function inside the referenced script; specifying only path expecting the whole script to be used; empty `preprocess:` key in YAML.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2f8ba64c3825d291. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_ml.py:100

        postprocess, 'postprocess') or self.default_postprocess_fn()

  def inference_output_type(self):
    return Any

  @staticmethod
  def parse_processing_transform(processing_transform, typ):
    def _parse_config(callable=None, path=None, name=None):
      if callable and (path or name):
        raise ValueError(
            f"Cannot specify 'callable' with 'path' and 'name' for {typ} "
            f"function.")
      if path and name:
        return python_callable.PythonCallableWithSource.load_from_script(
            FileSystems.open(path).read().decode(), name)
      elif callable:
        return python_callable.PythonCallableWithSource(callable)
      else:
        raise ValueError(
            f"Must specify one of 'callable' or 'path' and 'name' for {typ} "
            f"function.")

    if processing_transform:
      if isinstance(processing_transform, dict):
        return _parse_config(**processing_transform)
      else:
        raise ValueError("Invalid model_handler specification.")

  def underlying_handler(self):
    return self._handler

  @staticmethod
  def default_preprocess_fn():
    raise ValueError(
        'Model Handler does not implement a default preprocess '
        'method. Please define a preprocessing method using the '
        '\'preprocess\' tag. This is required in most cases because '

View on GitHub (pinned to 12126d8942)