What causes "Plugin '{name}' must subclass {base_class.__name__}" in p-e-w/heretic?

Make the class inherit from the expected base (e.g. class MyScorer(Scorer)).

p-e-w/heretic · error · TypeError

Plugin '{name}' must subclass {base_class.name}

Error message

Plugin '{name}' must subclass {base_class.__name__}

What it means

The plugin class was found and loaded, but issubclass(plugin_cls, base_class) failed: the class does not inherit from the required plugin base (e.g. Scorer). load_plugin() raises TypeError because loading a non-plugin class would break the evaluator contract downstream.

Source

Thrown at src/heretic/plugin.py:145

                sys.modules.pop(module_name, None)
                raise

        plugin_cls = validate_class(module, class_name)
    # Fully-qualified import path, e.g "heretic.scorers.keyword_rate.KeywordRate".
    else:
        if "." not in name:
            raise ValueError(
                "Import-based plugin must use the form 'fully.qualified.module.ClassName'"
            )
        module_name, class_name = name.rsplit(".", 1)
        try:
            module = importlib.import_module(module_name)
        except ImportError as e:
            raise ImportError(f"Error loading plugin '{name}': {e}") from e
        plugin_cls = validate_class(module, class_name)

    if not issubclass(plugin_cls, base_class):
        raise TypeError(f"Plugin '{name}' must subclass {base_class.__name__}")

    return plugin_cls


class Context:
    """
    Runtime context passed to plugins

    Provides plugin-safe access to the model.

    Plugins must use `get_responses(...)`, `get_logits(...)`, etc.
    Direct access to the underlying Model is intentionally not exposed.
    """

    def __init__(self, settings: HereticSettings, model: Model) -> None:
        self._model = model
        self._settings = settings
        self._responses_cache: dict[tuple[tuple[str, str], ...], list[str]] = {}

View on GitHub (pinned to bedb94ef11)

Solutions

Make the class inherit from the expected base (e.g. class MyScorer(Scorer)).
Point the plugin name at the class that actually subclasses the base.
After a heretic upgrade, update the plugin to the new base-class API and re-export it.

Example fix

# before
class MyScorer:
    ...
# after
from heretic.scorers import Scorer
class MyScorer(Scorer):
    ...

Defensive patterns

Strategy: type-guard

Validate before calling

from heretic.plugin import load_plugin
def check_plugin(name: str, base: type) -> bool:
    try:
        return issubclass(load_plugin(name, base), base)
    except (ValueError, ImportError, TypeError):
        return False

Type guard

def is_scorer(obj: object) -> bool:
    return inspect.isclass(obj) and issubclass(obj, Scorer)

Try / catch

try:
    cls = load_plugin(name, Scorer)
except TypeError as e:
    sys.exit(f"{e} — make sure your class subclasses Scorer")

Prevention

Always inherit from the required base class (Scorer, etc.).
After upgrading heretic, re-check your plugin against the current base-class API.
Export only the intended plugin class under the name you reference.

When it happens

Trigger: Passing 'mypkg.models.Thing' where Thing subclasses nothing relevant; a file plugin exporting a helper class instead of the plugin class; the base class changed between heretic versions so an old plugin no longer subclasses the new base (e.g. Plugin API refactor).

Common situations: Pointing the config at the wrong class in a module that defines several; plugin written for an older heretic release with a different base class; copy-paste left a plain class unmodified.

Related errors

AI-assisted analysis of p-e-w/heretic@bedb94ef11 (2026-08-29). Data as JSON: /api/errors/970427d74c23cd6e. Report an issue: GitHub.