{"record":{"id":"237078ab05647993","repo":"hiyouga/LlamaFactory","slug":"rm-trainer-currently-only-supports-cp-size-1","errorCode":null,"errorMessage":"RM trainer currently only supports cp_size == 1.","messagePattern":"RM trainer currently only supports cp_size == 1\\.","errorType":"validation","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/trainers/rm_trainer.py","lineNumber":80,"sourceCode":"        std = 1.0 / (hidden_size * 10)\n        with torch.no_grad():\n            score.weight.normal_(mean=0.0, std=std)\n            if score.bias is not None:\n                score.bias.zero_()\n        logger.info_rank0(f\"Initialized score head with Gaussian (std={std:.6f}): {score.weight.shape}\")\n\n\nclass RMTrainer(BaseTrainer):\n    def __init__(\n        self,\n        args: TrainingArguments,\n        model: HFModel,\n        renderer,\n        train_dataset,\n        callbacks=None,\n    ) -> None:\n        if args.cp_size > 1:\n            raise NotImplementedError(\"RM trainer currently only supports cp_size == 1.\")\n\n        super().__init__(args, model, renderer, train_dataset, callbacks)\n\n    def _shard_model(self) -> None:\n        if self.args.dist_config is None:\n            if DistributedInterface().get_world_size(Dim.DP) > 1:\n                from torch.nn.parallel import DistributedDataParallel as DDP\n\n                device_ids = None if self.device.type == \"cpu\" else [self.device.index]\n                self.model = DDP(self.model, device_ids=device_ids, find_unused_parameters=True)\n        else:\n            super()._shard_model()\n\n    @property\n    def _unwrapped_model(self):\n        \"\"\"Access the underlying model, unwrapping DDP/FSDP wrappers if present.\"\"\"\n        model = self.model\n        if hasattr(model, \"module\"):","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/trainers/rm_trainer.py#L62-L98","documentation":"NotImplementedError from RMTrainer.__init__ (rm_trainer.py:80) when args.cp_size > 1. The v1 reward-model trainer has no context-parallelism support; the guard fires before super().__init__() and before DDP/FSDP wrapping in _shard_model.","triggerScenarios":"Running RM training (stage rm) with cp_size > 1 in the v1 training arguments.","commonSituations":"Long preference sequences tempt users to enable CP to shard sequence length across GPUs; a cluster template sets cp_size globally and is reused for RM runs.","solutions":["Set cp_size: 1 for RM training.","Fit long sequences by raising memory headroom (activation checkpointing, smaller per-device batch, sequence truncation via cutoff_len) instead of CP.","If CP is mandatory, use the v0 RM trainer."],"exampleFix":"# before\ntraining_args:\n  cp_size: 4\n\n# after\ntraining_args:\n  cp_size: 1","handlingStrategy":"validation","validationCode":"if args.cp_size > 1:\n    raise SystemExit(\"v1 RMTrainer does not support cp_size > 1\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep cp_size at 1 for RM/DPO until the trainers advertise CP support.","Subscribe to release notes for v1 trainer capability changes."],"tags":["reward-model","distributed","context-parallelism","not-implemented"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}