apache/beam · error · ImportError

Could not import joblib in this execution environment. For h

Error message

Could not import joblib in this execution environment. For help with managing dependencies on Python workers.see https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/

What it means

Raised in sklearn_inference._load_model when a model was saved with joblib (ModelFileType.JOBLIB) but the joblib package is not importable in the worker's execution environment. The library deliberately checks `if not joblib` and raises an actionable ImportError pointing to Beam's dependency documentation.

Source

Thrown at sdks/python/apache_beam/ml/inference/sklearn_inference.py:63

]

NumpyInferenceFn = Callable[
    [BaseEstimator, Sequence[numpy.ndarray], Optional[dict[str, Any]]], Any]


class ModelFileType(enum.Enum):
  """Defines how a model file is serialized. Options are pickle or joblib."""
  PICKLE = 1
  JOBLIB = 2


def _load_model(model_uri, file_type):
  file = FileSystems.open(model_uri, 'rb')
  if file_type == ModelFileType.PICKLE:
    return pickle.load(file)
  elif file_type == ModelFileType.JOBLIB:
    if not joblib:
      raise ImportError(
          'Could not import joblib in this execution environment. '
          'For help with managing dependencies on Python workers.'
          'see https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/'  # pylint: disable=line-too-long
      )
    return joblib.load(file)
  raise AssertionError('Unsupported serialization type.')


def _default_numpy_inference_fn(
    model: BaseEstimator,
    batch: Sequence[numpy.ndarray],
    inference_args: Optional[dict[str, Any]] = None) -> Any:
  inference_args = {} if not inference_args else inference_args
  # vectorize data for better performance
  vectorized_batch = numpy.stack(batch, axis=0)
  return model.predict(vectorized_batch, **inference_args)

View on GitHub (pinned to 12126d8942)

Solutions

Add joblib to your requirements.txt or extra_package/dependency list so workers install it
Rebuild your custom container image so joblib is present in the worker environment
Switch to ModelFileType.PICKLE if joblib is not needed and the model was serialized with pickle

Example fix

// before
pipeline_options = PipelineOptions([])  # requirements.txt lacks joblib
handler = SklearnModelHandler(model_uri=uri, model_file_type=ModelFileType.JOBLIB)
// after
# requirements.txt: joblib
handler = SklearnModelHandler(model_uri=uri, model_file_type=ModelFileType.JOBLIB)

Defensive patterns

Strategy: validation

Validate before calling

def ensure_joblib_available(model_file_type):
    if model_file_type == ModelFileType.JOBLIB:
        import importlib.util
        if importlib.util.find_spec('joblib') is None:
            raise ImportError('joblib is required for JOBLIB model files; add it to worker dependencies.')

Try / catch

try:
    predictions = pcoll | RunInference(handler)
except ImportError as e:
    if 'joblib' in str(e):
        raise RuntimeError('Add joblib to requirements/container image for workers') from e
    raise

Prevention

Always include joblib in requirements.txt when using ModelFileType.JOBLIB
Test model loading in the same container image used by workers before launching the pipeline
Prefer pickle when joblib is not needed to reduce dependency surface

When it happens

Trigger: Creating SklearnModelHandler with model_file_type=ModelFileType.JOBLIB and loading a model on a worker where joblib is not installed or not shipped with the pipeline.

Common situations: Running the pipeline in a container/custom Docker image that includes scikit-learn inference code but not joblib; using --env_config or requirements_file that omitted joblib; a Flink/Dataflow worker using a base image without the extra dependency.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/77da5c399e0b1dfd. Report an issue: GitHub.