{"record":{"id":"77da5c399e0b1dfd","repo":"apache/beam","slug":"could-not-import-joblib-in-this-execution-environment-for","errorCode":null,"errorMessage":"Could not import joblib in this execution environment. For help with managing dependencies on Python workers.see https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/","messagePattern":"Could not import joblib in this execution environment\\. For help with managing dependencies on Python workers\\.see https://beam\\.apache\\.org/documentation/sdks/python-pipeline-dependencies/","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/inference/sklearn_inference.py","lineNumber":63,"sourceCode":"]\n\nNumpyInferenceFn = Callable[\n    [BaseEstimator, Sequence[numpy.ndarray], Optional[dict[str, Any]]], Any]\n\n\nclass ModelFileType(enum.Enum):\n  \"\"\"Defines how a model file is serialized. Options are pickle or joblib.\"\"\"\n  PICKLE = 1\n  JOBLIB = 2\n\n\ndef _load_model(model_uri, file_type):\n  file = FileSystems.open(model_uri, 'rb')\n  if file_type == ModelFileType.PICKLE:\n    return pickle.load(file)\n  elif file_type == ModelFileType.JOBLIB:\n    if not joblib:\n      raise ImportError(\n          'Could not import joblib in this execution environment. '\n          'For help with managing dependencies on Python workers.'\n          'see https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/'  # pylint: disable=line-too-long\n      )\n    return joblib.load(file)\n  raise AssertionError('Unsupported serialization type.')\n\n\ndef _default_numpy_inference_fn(\n    model: BaseEstimator,\n    batch: Sequence[numpy.ndarray],\n    inference_args: Optional[dict[str, Any]] = None) -> Any:\n  inference_args = {} if not inference_args else inference_args\n  # vectorize data for better performance\n  vectorized_batch = numpy.stack(batch, axis=0)\n  return model.predict(vectorized_batch, **inference_args)\n\n","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/inference/sklearn_inference.py#L45-L81","documentation":"Raised in sklearn_inference._load_model when a model was saved with joblib (ModelFileType.JOBLIB) but the joblib package is not importable in the worker's execution environment. The library deliberately checks `if not joblib` and raises an actionable ImportError pointing to Beam's dependency documentation.","triggerScenarios":"Creating SklearnModelHandler with model_file_type=ModelFileType.JOBLIB and loading a model on a worker where joblib is not installed or not shipped with the pipeline.","commonSituations":"Running the pipeline in a container/custom Docker image that includes scikit-learn inference code but not joblib; using --env_config or requirements_file that omitted joblib; a Flink/Dataflow worker using a base image without the extra dependency.","solutions":["Add joblib to your requirements.txt or extra_package/dependency list so workers install it","Rebuild your custom container image so joblib is present in the worker environment","Switch to ModelFileType.PICKLE if joblib is not needed and the model was serialized with pickle"],"exampleFix":"// before\npipeline_options = PipelineOptions([])  # requirements.txt lacks joblib\nhandler = SklearnModelHandler(model_uri=uri, model_file_type=ModelFileType.JOBLIB)\n// after\n# requirements.txt: joblib\nhandler = SklearnModelHandler(model_uri=uri, model_file_type=ModelFileType.JOBLIB)","handlingStrategy":"validation","validationCode":"def ensure_joblib_available(model_file_type):\n    if model_file_type == ModelFileType.JOBLIB:\n        import importlib.util\n        if importlib.util.find_spec('joblib') is None:\n            raise ImportError('joblib is required for JOBLIB model files; add it to worker dependencies.')","typeGuard":null,"tryCatchPattern":"try:\n    predictions = pcoll | RunInference(handler)\nexcept ImportError as e:\n    if 'joblib' in str(e):\n        raise RuntimeError('Add joblib to requirements/container image for workers') from e\n    raise","preventionTips":["Always include joblib in requirements.txt when using ModelFileType.JOBLIB","Test model loading in the same container image used by workers before launching the pipeline","Prefer pickle when joblib is not needed to reduce dependency surface"],"tags":["python","apache-beam","sklearn","dependency","import"],"backgroundTag":"missing-optional-dependency","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}