{"record":{"id":"341d6e0f7d29e0ee","repo":"sgl-project/sglang","slug":"dspark-aux-hidden-capture-requires-pp-1-341d6e","errorCode":null,"errorMessage":"DSPARK aux hidden capture requires PP=1.","messagePattern":"DSPARK aux hidden capture requires PP=1\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/kimi_linear.py","lineNumber":793,"sourceCode":"        if self.pp_group.is_last_rank:\n            self.lm_head = ParallelLMHead(\n                self.config.vocab_size,\n                self.config.hidden_size,\n                quant_config=quant_config,\n                prefix=maybe_prefix(prefix, \"lm_head\"),\n            )\n        else:\n            self.lm_head = PPMissingLayer()\n        logit_scale = getattr(self.config, \"logit_scale\", 1.0)\n        self.logits_processor = LogitsProcessor(config=config, logit_scale=logit_scale)\n        self.capture_aux_hidden_states = False\n\n    def get_input_embeddings(self):\n        return self.model.embed_tokens\n\n    def set_dspark_layers_to_capture(self, layer_ids: list[int]) -> None:\n        if self.pp_group.world_size > 1:\n            raise NotImplementedError(\"DSPARK aux hidden capture requires PP=1.\")\n        if not self.pp_group.is_last_rank:\n            return\n        if layer_ids is None:\n            raise ValueError(\n                \"DSPARK requires explicit layer_ids for aux hidden capture.\"\n            )\n        self.capture_aux_hidden_states = True\n        self.model.dspark_layers_to_capture = list(layer_ids)\n\n    @torch.no_grad()\n    def forward(\n        self,\n        input_ids: torch.Tensor,\n        positions: torch.Tensor,\n        forward_batch: ForwardBatch,\n        inputs_embeds: Optional[torch.Tensor] = None,\n        pp_proxy_tensors: Optional[PPProxyTensors] = None,\n    ) -> torch.Tensor:","sourceCodeStart":775,"sourceCodeEnd":811,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/kimi_linear.py#L775-L811","documentation":"KimiLinearModel.set_dspark_layers_to_capture (kimi_linear.py:793) enables DSPARK aux-hidden-state capture for training/distillation hooks. Aux states are gathered on the last pipeline stage only, so running with pipeline parallelism (pp_group.world_size > 1) is unsupported and raises NotImplementedError.","triggerScenarios":"Calling set_dspark_layers_to_capture while --pp-size (pipeline parallel) > 1, e.g. in a multi-node or memory-constrained PP deployment that also enables DSPARK capture.","commonSituations":"Large Kimi-Linear deployments that shard with PP for memory, then trying to attach DSPARK hidden-capture hooks.","solutions":["Run with PP=1 (use TP/EP for sharding instead) when DSPARK capture is needed","Disable the DSPARK capture feature / do not call the setter in PP deployments","Request/patch upstream support for gathering aux states across PP stages"],"exampleFix":"# before\npython -m sglang.launch_server --pp-size 2 ...  (+ dspark capture)\n# after\npython -m sglang.launch_server --pp-size 1 --tp-size 4 ...","handlingStrategy":"validation","validationCode":"assert model.pp_group.world_size == 1  # before set_dspark_layers_to_capture","typeGuard":null,"tryCatchPattern":"try:\n    model.set_dspark_layers_to_capture(ids)\nexcept NotImplementedError:\n    ids = None  # disable DSPARK under PP","preventionTips":["Decide PP vs feature hooks up front; they conflict","Document PP=1 requirement wherever DSPARK is enabled"],"tags":["kimi-linear","dspark","pipeline-parallel","not-implemented"],"backgroundTag":"parallelism-config-conflict","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}