mudler/LocalAI · error · ValueError

GRPO requires at least one reward function. Specify reward_f

Error message

GRPO requires at least one reward function. Specify reward_functions in the request or reward_funcs in extra_options.

What it means

GRPOTrainer in TRL is reward-driven: it cannot train without at least one reward function. The backend builds reward functions exclusively from the `reward_funcs` key in extra_options (parsed by build_reward_functions); if that key is absent, empty, or yields no functions, training aborts before the trainer is constructed.

Source

Thrown at backend/python/trl/backend.py:499

            num_generations = int(extra.get("num_generations", "4"))
            max_completion_length = int(extra.get("max_completion_length", "256"))

            training_args = GRPOConfig(
                num_generations=num_generations,
                max_completion_length=max_completion_length,
                **_common_args,
            )

            # GRPO requires reward functions passed via extra_options as a JSON list
            from reward_functions import build_reward_functions

            reward_funcs = []
            if extra.get("reward_funcs"):
                reward_funcs = build_reward_functions(extra["reward_funcs"])

            if not reward_funcs:
                raise ValueError(
                    "GRPO requires at least one reward function. "
                    "Specify reward_functions in the request or "
                    "reward_funcs in extra_options."
                )

            trainer = GRPOTrainer(
                model=model,
                args=training_args,
                train_dataset=dataset,
                processing_class=tokenizer,
                reward_funcs=reward_funcs,
                callbacks=[progress_cb.get_callback()],
            )

        elif training_method == "orpo":
            from trl import ORPOTrainer, ORPOConfig

            beta = float(extra.get("beta", "0.1"))

View on GitHub (pinned to 44413a9d06)

Solutions

1. Add reward_funcs to extra_options as a JSON array, e.g. extra_options={'reward_funcs': '[{"type":"builtin","name":"format_reward"}]'}.
2. Verify the key is exactly 'reward_funcs' in extra_options and the JSON parses to a non-empty list.
3. For custom rewards, provide an inline spec (requires LOCALAI_TRL_ALLOW_INLINE_REWARD=true) or a builtin name from the registry.

Example fix

# before
extra = {}  # GRPO with no rewards
# after
extra = {"reward_funcs": json.dumps([{ "type": "builtin", "name": "format_reward" }])}

def grpo_request_valid(extra_options: dict) -> bool:
    specs = extra_options.get("reward_funcs")
    if not specs:
        return False
    import json
    parsed = json.loads(specs) if isinstance(specs, str) else specs
    return isinstance(parsed, list) and len(parsed) > 0

try:
    run_finetune(req)
except ValueError as e:
    if "reward function" in str(e):
        req.extra_options["reward_funcs"] = '[{"type":"builtin","name":"format_reward"}]'
        run_finetune(req)
    else:
        raise

When it happens

Trigger: Sending a FineTune request with training_method='grpo' but no reward_funcs in extra_options; passing reward_funcs='[]' (empty JSON array); misspelling the key (e.g. 'reward_functions' — note the error text mentions it but the code reads extra['reward_funcs']).

Common situations: Copy-pasting a GRPO example without the reward section; assuming the dataset's label column is used as reward (it is not, in GRPO); key-name confusion between the protobuf field and the extra_options key.

Related errors

• Dataset source path is outside the allowed directory
• Unsupported training method: {training_method}. Supported: s
• Syntax error in inline reward function '{name}': {e}
• Inline reward function '{name}' must return a list, got {typ
• reward_funcs must be a JSON array of reward function specs

AI-assisted analysis of mudler/LocalAI@44413a9d06 (2026-08-15). Data as JSON: /api/errors/e3e0bcc27393dd9e. Report an issue: GitHub.