unclecode/crawl4ai · error · ValueError

Cannot extract source code for hook '{hook_name}'. Make sure

Error message

Cannot extract source code for hook '{hook_name}'. Make sure the function is defined in a file (not interactively). Error: {e}

What it means

hooks_to_string (crawl4ai/utils.py:3735) raises ValueError when inspect.getsource(hook_func) raises OSError/TypeError. getsource can only retrieve source for functions defined in a real file that the interpreter can locate — lambdas passed directly, functions defined in the REPL/Jupyter (partially), C builtins, or objects whose source file is gone/moved all fail here. The function needs the literal source text because it ships hooks to a Docker-based crawl4ai API.

Source

Thrown at crawl4ai/utils.py:3735

        >>> # api_hooks is now ready to use with Docker API

    Raises:
        ValueError: If a hook is not callable or source cannot be extracted
    """
    result = {}

    for hook_name, hook_func in hooks.items():
        if not callable(hook_func):
            raise ValueError(f"Hook '{hook_name}' must be a callable function, got {type(hook_func)}")

        try:
            # Get the source code of the function
            source = inspect.getsource(hook_func)
            # Remove any leading indentation to get clean source
            source = textwrap.dedent(source)
            result[hook_name] = source
        except (OSError, TypeError) as e:
            raise ValueError(
                f"Cannot extract source code for hook '{hook_name}'. "
                f"Make sure the function is defined in a file (not interactively). Error: {e}"
            )

    return result

View on GitHub (pinned to 7e80152142)

Solutions

Replace lambdas with module-level def functions so inspect.getsource can find them.
Ensure the hook lives in an importable .py file on disk that stays in place during the call.
Avoid functools.partial/builtins as hooks — write a plain def that forwards to them.
If you must generate hooks dynamically, write them to a temp .py file, import it, then pass that function.

Example fix

# before
hooks_to_string({"on_page_context_created": lambda ctx: None})

# after
# hooks.py
def on_page_context_created(page, context, **kwargs):
    return None

from hooks import on_page_context_created
hooks_to_string({"on_page_context_created": on_page_context_created})

Defensive patterns

Strategy: type-guard

Validate before calling

import inspect

def hook_source_extractable(func) -> bool:
    if not callable(func):
        return False
    try:
        inspect.getsource(func)
        return True
    except (OSError, TypeError):
        return False

Type guard

import inspect, types

def is_serializable_hook(func) -> bool:
    return (
        callable(func)
        and not isinstance(func, types.BuiltinFunctionType)
        and getattr(func, "__name__", "<lambda>") != "<lambda>"
        and hook_source_extractable(func)
    )

Try / catch

try:
    api_hooks = hooks_to_string(hooks)
except ValueError as e:
    if "source code" in str(e):
        raise ValueError(f"Move hook to a module-level def in a .py file: {e}")
    raise

Prevention

Never use lambdas or partials as serializable hooks
Keep hooks in stable .py files present for the process lifetime
For dynamic hooks, generate a temp module and import it before serialization

When it happens

Trigger: Using a lambda as a hook: hooks_to_string({"on_url_context": lambda ctx: None}); defining the hook interactively in ipython; decorating with a wrapper whose module file was deleted after import; passing functools.partial objects or callables implemented in C.

Common situations: Quick notebook experiments with lambda hooks; hooks defined inside exec'd dynamic code; source tree modified after the process imported the module (linecache miss).

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/fb6505f368d59b61. Report an issue: GitHub.