{"record":{"id":"0a3b9855ac2cebbb","repo":"sgl-project/sglang","slug":"all-memory-audio-latents-must-share-batch-and-chan","errorCode":null,"errorMessage":"All memory audio latents must share batch and channel dimensions","messagePattern":"All memory audio latents must share batch and channel dimensions","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/joy_echo/memory.py","lineNumber":835,"sourceCode":"        self._trim()\n        return metadata\n\n    def get_memory_frames(self) -> list[Image.Image | list[Image.Image]]:\n        return [entry.frame for entry in self.memory]\n\n    def get_memory_audio(self) -> Optional[torch.Tensor]:\n        audio_latents = [entry.audio_latent for entry in self.memory]\n        if not audio_latents or any(item is None for item in audio_latents):\n            return None\n        first = audio_latents[0]\n        assert first is not None\n        for audio_latent in audio_latents:\n            assert audio_latent is not None\n            if (\n                audio_latent.shape[0] != first.shape[0]\n                or audio_latent.shape[2] != first.shape[2]\n            ):\n                raise ValueError(\n                    \"All memory audio latents must share batch and channel dimensions\"\n                )\n        return torch.cat(audio_latents, dim=1).contiguous()\n\n    def get_memory_audio_segment_lengths(self) -> tuple[tuple[int, ...], ...]:\n        audio_latents = [entry.audio_latent for entry in self.memory]\n        if not audio_latents or any(item is None for item in audio_latents):\n            return ()\n        return (\n            tuple(\n                int(audio_latent.shape[1])\n                for audio_latent in audio_latents\n                if audio_latent is not None\n            ),\n        )\n\n    def __len__(self) -> int:\n        return len(self.memory)","sourceCodeStart":817,"sourceCodeEnd":853,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/joy_echo/memory.py#L817-L853","documentation":"When multiple memory entries are retrieved, get_memory_audio concatenates their audio latents along the time dim (dim=1) and requires every latent to share the same batch (dim 0) and channel (dim 2) sizes. Mismatched codecs, channel counts, or batch dims would make the concatenated tensor ill-formed, so it raises before torch.cat.","triggerScenarios":"Calling get_memory_audio after saving slots with latents from different audio codecs or channel widths (e.g. one [1, T, 64] and another [1, T, 128]), or after saving one latent with batch 1 and another with batch 2.","commonSituations":"Mixing latents from different model versions/checkpoints in one memory bank; saving a multi-batch latent via a path that bypassed validation; changing audio codec config between saves in a long-running session.","solutions":["Re-encode all memory audio with the same codec/config so latents share B and C dims; rebuild the memory bank","Audit each saved entry's audio_latent.shape and evict/re-save the outliers","Ensure every save path goes through _prepare_audio_latent and enforces B==1"],"exampleFix":"# before\nbank.save_memory_slot(\"a\", audio_latent=lat_64ch)   # [1, T, 64]\nbank.save_memory_slot(\"b\", audio_latent=lat_128ch) # [1, T, 128]\nmerged = bank.get_memory_audio()\n# after\nbank.save_memory_slot(\"a\", audio_latent=encode(audio))  # same codec for all\nbank.save_memory_slot(\"b\", audio_latent=encode(audio))\nmerged = bank.get_memory_audio()","handlingStrategy":"validation","validationCode":"shapes = {(e.audio_latent.shape[0], e.audio_latent.shape[2]) for e in bank.memory if e.audio_latent is not None}\nif len(shapes) > 1:\n    raise ValueError(f\"memory bank has inconsistent audio latent dims: {shapes}; re-encode all slots with one codec\")","typeGuard":"def bank_audio_is_consistent(bank) -> bool:\n    shapes = {(e.audio_latent.shape[0], e.audio_latent.shape[2]) for e in bank.memory if e.audio_latent is not None}\n    return len(shapes) <= 1","tryCatchPattern":null,"preventionTips":["Encode all memory audio with one codec/config; rebuild the bank when the codec changes","Assert audio_latent.shape[0]==1 on every save so batch dims can never diverge"],"tags":["joy-echo","audio","latent","memory-bank","concat"],"backgroundTag":"shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}