apache/beam · error · ValueError

Unable to import HuggingFacePipelineModelHandler. Please…

Error message

Unable to import HuggingFacePipelineModelHandler. Please install transformers dependencies.

What it means

Raised when HuggingFacePipelineModelHandler cannot be imported because the transformers (and related torch) dependencies are not installed. yaml_ml imports it lazily in __init__ and re-raises the ImportError as a ValueError with guidance.

Solutions

  1. pip install apache_beam[transformers] (or pip install transformers torch)
  2. Add transformers and torch to requirements/worker packages passed to the runner
  3. Verify the import works in the actual execution environment, not just locally

Example fix

# before
pip install apache_beam
# after
pip install 'apache_beam[transformers]'
Defensive patterns

Strategy: try-catch

Validate before calling

try:
    from apache_beam.ml.inference.huggingface_inference import HuggingFacePipelineModelHandler
except ImportError:
    raise SystemExit("Run: pip install 'apache_beam[transformers]'")

Type guard

def has_hf_handler():
    import importlib.util
    return importlib.util.find_spec('apache_beam.ml.inference.huggingface_inference') is not None

Try / catch

try:
    transform = RunInference(model_handler=hf_spec)
except ValueError as e:
    if 'transformers dependencies' in str(e):
        install_transformers()
    else:
        raise

Prevention

When it happens

Trigger: Configuring a YAML RunInference transform with a HuggingFace pipeline handler while apache_beam was installed without the transformers extra, e.g. missing `apache_beam[transformers]` or a standalone install of transformers/torch.

Common situations: Running Beam YAML pipelines in slim containers; forgetting to add transformers/torch to worker requirements; CPU-only environments where torch failed to install; Airflow/CI runners without ML deps.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3de0f2b0485ea8b7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_ml.py:343

      model: The model name on Hugging Face hub or a path to a local directory.
        If the model already defines the task, no need to specify the task.
      preprocess: A python callable, defined either inline, or using a file,
        that is invoked on the input row before sending to the model to be
        loaded by this ModelHandler.
      postprocess: A python callable, defined either inline, or using a file,
        that is invoked on the PredictionResult output by the ModelHandler
        before parsing into the output Beam Row.
      device: The device to run the pipeline on (e.g., 'cpu', 'cuda', 'cuda:0').
        Defaults to CPU.
      inference_fn: The custom inference function to use.
      load_pipeline_args: Extra arguments to pass to the Hugging Face pipeline
        loader (e.g. `transformers.pipeline`).
      **kwargs: Extra arguments to pass to the model handler.
    """
    try:
      from apache_beam.ml.inference.huggingface_inference import HuggingFacePipelineModelHandler
    except ImportError:
      raise ValueError(
          'Unable to import HuggingFacePipelineModelHandler. Please '
          'install transformers dependencies.')

    kwargs = {k: v for k, v in kwargs.items() if not k.startswith('_')}

    inference_fn_obj = self.parse_processing_transform(
        inference_fn, 'inference_fn') if inference_fn else None

    handler_kwargs = {}
    if inference_fn_obj:
      handler_kwargs['inference_fn'] = inference_fn_obj

    _handler = HuggingFacePipelineModelHandler(
        task=task,
        model=model,
        device=device,
        load_pipeline_args=load_pipeline_args,
        **handler_kwargs,

View on GitHub (pinned to 12126d8942)