apache/beam · error · ValueError

Cannot specify 'callable' with 'path' and 'name' for

Error message

Cannot specify 'callable' with 'path' and 'name' for {typ} function.

What it means

Beam YAML ML's `parse_processing_transform` config parser (`_parse_config`, yaml_ml.py:91) accepts either an inline `callable` OR a `path`+`name` pair referencing a script, but not both. Specifying both is ambiguous, so a ValueError is raised.

Solutions

Remove the `path`/`name` keys and keep only `callable` if you want an inline function.
Remove `callable` and keep `path` + `name` to load from a script file.
Validate the transform config dict before passing it so only one form is present.

Example fix

# before
preprocess: {callable: my_fn, path: preprocess.py, name: my_fn}
# after
preprocess: {path: preprocess.py, name: my_fn}

Defensive patterns

Strategy: validation

Validate before calling

cfg = processing_transform if isinstance(processing_transform, dict) else {}
if 'callable' in cfg and ('path' in cfg or 'name' in cfg):
    raise ValueError('use either callable or path+name, not both')

Type guard

def has_conflicting_fn_config(cfg):
    return 'callable' in cfg and bool(cfg.get('path') or cfg.get('name'))

Try / catch

try:
    fn = parse_processing_transform(spec, typ)
except ValueError as e:
    raise YamlConfigError(f'preprocess config invalid: {e}') from e

Prevention

Choose one form: inline callable OR script path+name, never both.
When editing configs, remove the alternative form's keys entirely.
Validate preprocess blocks with a schema check before submitting the pipeline.

When it happens

Trigger: Calling parse_processing_transform / configuring a ML transform with a dict containing both a `callable` and a `path` (or `name`) key, e.g. {callable: my_fn, path: preprocess.py}.

Common situations: Copying a config template that had path/name and pasting an inline callable on top; merging two preprocess configs; leaving a leftover `path` key when switching to inline callables.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/98dd53eab595da4a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_ml.py:91

  def __init__(
      self,
      handler,
      preprocess: Optional[dict[str, str]] = None,
      postprocess: Optional[dict[str, str]] = None):
    self._handler = handler
    self._preprocess_fn = self.parse_processing_transform(
        preprocess, 'preprocess') or self.default_preprocess_fn()
    self._postprocess_fn = self.parse_processing_transform(
        postprocess, 'postprocess') or self.default_postprocess_fn()

  def inference_output_type(self):
    return Any

  @staticmethod
  def parse_processing_transform(processing_transform, typ):
    def _parse_config(callable=None, path=None, name=None):
      if callable and (path or name):
        raise ValueError(
            f"Cannot specify 'callable' with 'path' and 'name' for {typ} "
            f"function.")
      if path and name:
        return python_callable.PythonCallableWithSource.load_from_script(
            FileSystems.open(path).read().decode(), name)
      elif callable:
        return python_callable.PythonCallableWithSource(callable)
      else:
        raise ValueError(
            f"Must specify one of 'callable' or 'path' and 'name' for {typ} "
            f"function.")

    if processing_transform:
      if isinstance(processing_transform, dict):
        return _parse_config(**processing_transform)
      else:
        raise ValueError("Invalid model_handler specification.")

View on GitHub (pinned to 12126d8942)