{"record":{"id":"e3e0bcc27393dd9e","repo":"mudler/LocalAI","slug":"grpo-requires-at-least-one-reward-function-specif","errorCode":null,"errorMessage":"GRPO requires at least one reward function. Specify reward_functions in the request or reward_funcs in extra_options.","messagePattern":"GRPO requires at least one reward function\\. Specify reward_functions in the request or reward_funcs in extra_options\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"backend/python/trl/backend.py","lineNumber":499,"sourceCode":"\n            num_generations = int(extra.get(\"num_generations\", \"4\"))\n            max_completion_length = int(extra.get(\"max_completion_length\", \"256\"))\n\n            training_args = GRPOConfig(\n                num_generations=num_generations,\n                max_completion_length=max_completion_length,\n                **_common_args,\n            )\n\n            # GRPO requires reward functions passed via extra_options as a JSON list\n            from reward_functions import build_reward_functions\n\n            reward_funcs = []\n            if extra.get(\"reward_funcs\"):\n                reward_funcs = build_reward_functions(extra[\"reward_funcs\"])\n\n            if not reward_funcs:\n                raise ValueError(\n                    \"GRPO requires at least one reward function. \"\n                    \"Specify reward_functions in the request or \"\n                    \"reward_funcs in extra_options.\"\n                )\n\n            trainer = GRPOTrainer(\n                model=model,\n                args=training_args,\n                train_dataset=dataset,\n                processing_class=tokenizer,\n                reward_funcs=reward_funcs,\n                callbacks=[progress_cb.get_callback()],\n            )\n\n        elif training_method == \"orpo\":\n            from trl import ORPOTrainer, ORPOConfig\n\n            beta = float(extra.get(\"beta\", \"0.1\"))","sourceCodeStart":481,"sourceCodeEnd":517,"githubUrl":"https://github.com/mudler/LocalAI/blob/44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26/backend/python/trl/backend.py#L481-L517","documentation":"GRPOTrainer in TRL is reward-driven: it cannot train without at least one reward function. The backend builds reward functions exclusively from the `reward_funcs` key in extra_options (parsed by build_reward_functions); if that key is absent, empty, or yields no functions, training aborts before the trainer is constructed.","triggerScenarios":"Sending a FineTune request with training_method='grpo' but no reward_funcs in extra_options; passing reward_funcs='[]' (empty JSON array); misspelling the key (e.g. 'reward_functions' — note the error text mentions it but the code reads extra['reward_funcs']).","commonSituations":"Copy-pasting a GRPO example without the reward section; assuming the dataset's label column is used as reward (it is not, in GRPO); key-name confusion between the protobuf field and the extra_options key.","solutions":["Add reward_funcs to extra_options as a JSON array, e.g. extra_options={'reward_funcs': '[{\"type\":\"builtin\",\"name\":\"format_reward\"}]'}.","Verify the key is exactly 'reward_funcs' in extra_options and the JSON parses to a non-empty list.","For custom rewards, provide an inline spec (requires LOCALAI_TRL_ALLOW_INLINE_REWARD=true) or a builtin name from the registry."],"exampleFix":"# before\nextra = {}  # GRPO with no rewards\n# after\nextra = {\"reward_funcs\": json.dumps([{ \"type\": \"builtin\", \"name\": \"format_reward\" }])}","handlingStrategy":"validation","validationCode":"def grpo_request_valid(extra_options: dict) -> bool:\n    specs = extra_options.get(\"reward_funcs\")\n    if not specs:\n        return False\n    import json\n    parsed = json.loads(specs) if isinstance(specs, str) else specs\n    return isinstance(parsed, list) and len(parsed) > 0","typeGuard":null,"tryCatchPattern":"try:\n    run_finetune(req)\nexcept ValueError as e:\n    if \"reward function\" in str(e):\n        req.extra_options[\"reward_funcs\"] = '[{\"type\":\"builtin\",\"name\":\"format_reward\"}]'\n        run_finetune(req)\n    else:\n        raise","preventionTips":["Always pair training_method='grpo' with a non-empty reward_funcs array in request builders.","Unit-test the extra_options payload shape before sending.","Keep a snippet library of valid reward specs."],"tags":["trl","grpo","fine-tuning","reward-functions","localai"],"backgroundTag":null,"analyzedSha":"44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26","analyzedAt":"2026-08-15T10:13:50.291Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}