run-llama/llama_index · error · ImportError

DeepEvalCallbackHandler is not installed. Please install it

Error message

DeepEvalCallbackHandler is not installed. Please install it using `pip install llama-index-callbacks-deepeval`

What it means

Raised by create_global_handler() when eval_mode is 'deepeval' but llama-index-callbacks-deepeval is not installed. The DeepEval callback (LLM evaluation metrics) is optional and imported lazily; a missing wheel produces this ImportError naming the package.

Source

Thrown at llama-index-core/llama_index/core/callbacks/global_handlers.py:84

        handler = honeyhive_callback_handler(**eval_params)
    elif eval_mode == "promptlayer":
        try:
            from llama_index.callbacks.promptlayer import (
                PromptLayerHandler,
            )  # pants: no-infer-dep
        except ImportError:
            raise ImportError(
                "PromptLayerHandler is not installed. "
                "Please install it using `pip install llama-index-callbacks-promptlayer`"
            )
        handler = PromptLayerHandler(**eval_params)
    elif eval_mode == "deepeval":
        try:
            from llama_index.callbacks.deepeval import (
                deepeval_callback_handler,
            )  # pants: no-infer-dep
        except ImportError:
            raise ImportError(
                "DeepEvalCallbackHandler is not installed. "
                "Please install it using `pip install llama-index-callbacks-deepeval`"
            )
        handler = deepeval_callback_handler(**eval_params)
    elif eval_mode == "simple":
        handler = SimpleLLMHandler(**eval_params)
    elif eval_mode == "argilla":
        try:
            from llama_index.callbacks.argilla import (
                argilla_callback_handler,
            )  # pants: no-infer-dep
        except ImportError:
            raise ImportError(
                "ArgillaCallbackHandler is not installed. "
                "Please install it using `pip install llama-index-callbacks-argilla`"
            )
        handler = argilla_callback_handler(**eval_params)
    elif eval_mode == "langfuse":

View on GitHub (pinned to afd0fef371)

Solutions

pip install llama-index-callbacks-deepeval
Verify: python -c "from llama_index.callbacks.deepeval import deepeval_callback_handler"
Pin the integration version in requirements.txt
Unset the deepeval eval mode for runs that do not evaluate

Example fix

# before
set_global_handler('deepeval')  # ImportError

# after
# pip install llama-index-callbacks-deepeval
set_global_handler('deepeval')

Defensive patterns

Strategy: validation

Validate before calling

import importlib.util
if importlib.util.find_spec('llama_index.callbacks.deepeval') is None:
    raise SystemExit('pip install llama-index-callbacks-deepeval')

Try / catch

try:
    handler = create_global_handler('deepeval')
except ImportError as e:
    raise  # evaluation runs should fail loudly if deepeval is missing

Prevention

Separate eval dependencies into an extras group (e.g. -e '.[eval]') so eval jobs fail at install, not runtime
Smoke-test eval entry points in CI

When it happens

Trigger: create_global_handler('deepeval', ...) / set_global_handler('deepeval') in an env without the wheel; `from llama_index.callbacks.deepeval import deepeval_callback_handler` raises ImportError.

Common situations: Running evaluation-enabled scripts on machines where only llama-index-core was pinned; CI pipelines that cache an older dependency set; installing deepeval itself but not the LlamaIndex bridge package.

Related errors

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/4e090a67c39a334b. Report an issue: GitHub.