p-e-w/heretic · error · ValueError

Duplicate scorer instance name: {scorer_key}. Give each inst

Error message

Duplicate scorer instance name: {scorer_key}. Give each instance a unique `instance_name`.

What it means

Each scorer instance gets a key of `<PluginClassName>_<instance_name>` (or just the class name if no instance_name). Two configurations resolving to the same key would silently overwrite each other's settings/state, so a duplicate key raises with instructions to set a unique instance_name.

Source

Thrown at src/heretic/evaluator.py:90

            )
            scorer_settings: BaseModel | None = scorer_cls.validate_settings(
                raw_settings
            )

            scorer = scorer_cls(
                heretic_settings=self.settings,
                settings=scorer_settings,
            )

            # External labeling key: ensures multiple instances can coexist.
            # Uses underscore to match the TOML namespace format (`scorer.<Class>_<instance>`).
            scorer_key = (
                scorer_cls.__name__
                if not instance_name
                else f"{scorer_cls.__name__}_{instance_name}"
            )
            if scorer_key in scorer_keys:
                raise ValueError(
                    f"Duplicate scorer instance name: {scorer_key}. "
                    "Give each instance a unique `instance_name`."
                )
            scorer_keys.add(scorer_key)

            scorer_instance_name = (
                f"{scorer.score_name} - {instance_name}"
                if instance_name
                else scorer.score_name
            )
            self._scorer_entries.append(
                ScorerEntry(scorer=scorer, config=config, name=scorer_instance_name)
            )

        # Run scorer init hooks.
        ctx = Context(settings=self.settings, model=self.model)

        for entry in self._scorer_entries:

View on GitHub (pinned to bedb94ef11)

Solutions

1. Give each duplicate plugin entry a distinct instance_name (e.g. "strict", "lenient")
2. Remove the redundant duplicate [[scorers]] entry if it was accidental
3. Key the key derivation by also distinguishing settings if you intend genuinely separate instances

Example fix

// before (config.toml)
[[scorers]]
plugin = "refusal"
[[scorers]]
plugin = "refusal"
// after (config.toml)
[[scorers]]
plugin = "refusal"
instance_name = "strict"
[[scorers]]
plugin = "refusal"
instance_name = "lenient"

keys = [f"{s.plugin}_{s.instance_name}" if s.instance_name else s.plugin for s in settings.scorers]
if len(keys) != len(set(keys)):
    raise ValueError(f"Duplicate scorer keys: {keys}")

try:
    evaluator = Evaluator(settings)
except ValueError as e:
    if e.message.startswith("Duplicate scorer instance name"):
        dedupe_scorer_configs(settings)
        evaluator = Evaluator(settings)
    else:
        raise

When it happens

Trigger: Configuring the same scorer plugin twice without distinct instance_name values (e.g. two [[scorers]] entries with plugin = "refusal" and no instance_name), or two entries whose class names and instance names collide.

Common situations: Running one scorer with different settings copies (forgetting to rename instances), refactoring a config and duplicating an entry, or copying an existing [[scorers]] block to add a variant.

Related errors

• '.' is not allowed
• whitespace is not allowed
• cannot be empty or whitespace
• No scorers configured. Set 'scorers' in config.toml
• Plugin namespace [{namespace}] must be a table/object, got {

AI-assisted analysis of p-e-w/heretic@bedb94ef11 (2026-08-29). Data as JSON: /api/errors/1fc9ca2073b887c7. Report an issue: GitHub.