{"record":{"id":"014d775d8a6ab50c","repo":"hiyouga/LlamaFactory","slug":"invalid-role","errorCode":null,"errorMessage":"Invalid role","messagePattern":"Invalid role","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"src/llamafactory/api/chat.py","lineNumber":105,"sourceCode":"        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=\"Invalid length\")\n\n    if request.messages[0].role == Role.SYSTEM:\n        content = request.messages.pop(0).content\n        if isinstance(content, list):\n            system = content[0].text if content else \"\"\n        else:\n            system = content\n    else:\n        system = None\n\n    if len(request.messages) % 2 == 0:\n        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=\"Only supports u/a/u/a/u...\")\n\n    input_messages = []\n    images, videos, audios = [], [], []\n    for i, message in enumerate(request.messages):\n        if i % 2 == 0 and message.role not in [Role.USER, Role.TOOL]:\n            raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=\"Invalid role\")\n        elif i % 2 == 1 and message.role not in [Role.ASSISTANT, Role.FUNCTION]:\n            raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=\"Invalid role\")\n\n        if message.role == Role.ASSISTANT and isinstance(message.tool_calls, list) and len(message.tool_calls):\n            tool_calls = [\n                {\"name\": tool_call.function.name, \"arguments\": tool_call.function.arguments}\n                for tool_call in message.tool_calls\n            ]\n            content = json.dumps(tool_calls, ensure_ascii=False)\n            input_messages.append({\"role\": ROLE_MAPPING[Role.FUNCTION], \"content\": content})\n        elif isinstance(message.content, list):\n            text_content = \"\"\n            for input_item in message.content:\n                if input_item.type == \"text\":\n                    text_content += input_item.text\n                elif input_item.type == \"image_url\":\n                    text_content += IMAGE_PLACEHOLDER\n                    image_url = input_item.image_url.url","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/api/chat.py#L87-L123","documentation":"LlamaFactory's PPO trainer subclasses TRL 0.9.x's PPOTrainer, whose training loop never supported evaluation datasets; run_ppo therefore rejects eval_dataset up front (trainer.py:84) with NotImplementedError instead of silently ignoring or crashing mid-eval. This is a design limitation of the pinned TRL version, not a transient bug.","triggerScenarios":"A PPO YAML that sets `val_size: >0` / eval_dataset (e.g. copied from an SFT config with eval split), causing run_ppo to construct PPOTrainer with eval_dataset not None.","commonSituations":"Copying an SFT/DPO config with val_size for PPO; assuming all stages share the eval interface.","solutions":["Remove eval settings from the PPO config: set `val_size: 0` and drop eval_dataset entries","If periodic quality checks are needed, run separate inference/eval jobs (llamafactory-cli chat / eval) on checkpoints instead of in-training eval"],"exampleFix":"# before (YAML)\nstage: ppo\nval_size: 0.1  # creates eval_dataset -> NotImplementedError\n\n# after\nstage: ppo\nval_size: 0","handlingStrategy":"validation","validationCode":"assert data_args.val_size == 0 and not data_args.eval_dataset, (\n    'PPOTrainer does not support eval datasets; remove val_size/eval_dataset from PPO configs'\n)","typeGuard":"def ppo_config_is_eval_free(yaml_cfg: dict) -> bool:\n    return float(yaml_cfg.get('val_size', 0) or 0) == 0 and not yaml_cfg.get('eval_dataset')","tryCatchPattern":"try:\n    from llamafactory.train.ppo.workflow import run_ppo\n    run_ppo(train_args)\nexcept NotImplementedError as e:\n    if 'eval dataset' in str(e):\n        raise SystemExit('Remove val_size/eval_dataset from the PPO config') from e\n    raise","preventionTips":["Keep stage-specific config templates; don't copy SFT configs wholesale into PPO","Evaluate PPO checkpoints with offline eval jobs instead of in-training eval"],"tags":["ppo","rlhf","eval-dataset","config","not-implemented"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}