p-e-w/heretic · critical · ValueError

No scorers configured. Set 'scorers' in config.toml

Error message

No scorers configured. Set 'scorers' in config.toml

What it means

The Evaluator requires at least one scorer plugin to judge model outputs during evaluation. If settings.scorers is empty or missing, evaluation would be meaningless, so the constructor raises immediately with guidance to set the 'scorers' table in config.toml.

Source

Thrown at src/heretic/evaluator.py:54

        self.model = model
        self._scorer_entries: list[ScorerEntry] = []

        print()
        print("Loading and initializing scorers...")
        self._load_and_init_scorers()

        # Establish baseline scores (pre-abliteration).
        self.baseline_scores = self.get_baseline_scores()
        self._print_baseline()

    def _load_and_init_scorers(self) -> None:
        """
        Load and instantiate all configured scorer plugins,
        then runs their initialization hooks.
        """
        scorer_configs = self.settings.scorers
        if not scorer_configs:
            raise ValueError("No scorers configured. Set 'scorers' in config.toml")

        scorer_keys: set[str] = set()

        # Resolve plugin classes from names and validate.
        for config in scorer_configs:
            scorer_cls = load_plugin(name=config.plugin, base_class=Scorer)
            scorer_cls.validate_contract()

            print(
                f"* Loaded: [bold]{scorer_cls.__name__} {'- ' + config.instance_name if config.instance_name else ''}[/bold]"
            )

            # Instantiate scorers.
            instance_name = config.instance_name or None

            raw_settings = self._get_scorer_settings_raw(
                scorer_cls=scorer_cls, instance_name=instance_name
            )

View on GitHub (pinned to bedb94ef11)

Solutions

1. Add a [scorers] section to config.toml with at least one scorer entry specifying its plugin
2. If using Settings programmatically, pass scorers=[ScorerConfig(plugin="...")].
3. Verify you are loading the intended config file (check config path/working directory)

Example fix

// before (config.toml)
# no [scorers] section
// after (config.toml)
[[scorers]]
plugin = "refusal"

if not getattr(settings, "scorers", None):
    raise ValueError("Configure at least one [[scorers]] entry in config.toml before creating an Evaluator")

try:
    evaluator = Evaluator(settings)
except ValueError as e:
    if "No scorers configured" in str(e):
        settings.scorers = [ScorerConfig(plugin="refusal")]
        evaluator = Evaluator(settings)
    else:
        raise

When it happens

Trigger: Constructing Evaluator(settings) where settings.scorers is None or an empty list — e.g. config.toml has no [scorers] section, or scorers was explicitly set to [].

Common situations: New projects started from a minimal config.toml, config files where the [scorers] section was commented out, or programmatic Settings() construction that omitted the scorers field.

Understand the failure class

Background: "X is required", "must be set", "cannot be empty": the missing-required-config error family, from Vertex AI project/location to WeChat keys — this error's family across 18 libraries.

Related errors

• cannot be empty or whitespace
• '.' is not allowed
• whitespace is not allowed
• Duplicate scorer instance name: {scorer_key}. Give each inst
• Plugin namespace [{namespace}] must be a table/object, got {

AI-assisted analysis of p-e-w/heretic@bedb94ef11 (2026-08-29). Data as JSON: /api/errors/b63147d92459fcb2. Report an issue: GitHub.