sgl-project/sglang · error · ValueError

All memory audio latents must share batch and channel dimens

Error message

All memory audio latents must share batch and channel dimensions

What it means

When multiple memory entries are retrieved, get_memory_audio concatenates their audio latents along the time dim (dim=1) and requires every latent to share the same batch (dim 0) and channel (dim 2) sizes. Mismatched codecs, channel counts, or batch dims would make the concatenated tensor ill-formed, so it raises before torch.cat.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/joy_echo/memory.py:835

        self._trim()
        return metadata

    def get_memory_frames(self) -> list[Image.Image | list[Image.Image]]:
        return [entry.frame for entry in self.memory]

    def get_memory_audio(self) -> Optional[torch.Tensor]:
        audio_latents = [entry.audio_latent for entry in self.memory]
        if not audio_latents or any(item is None for item in audio_latents):
            return None
        first = audio_latents[0]
        assert first is not None
        for audio_latent in audio_latents:
            assert audio_latent is not None
            if (
                audio_latent.shape[0] != first.shape[0]
                or audio_latent.shape[2] != first.shape[2]
            ):
                raise ValueError(
                    "All memory audio latents must share batch and channel dimensions"
                )
        return torch.cat(audio_latents, dim=1).contiguous()

    def get_memory_audio_segment_lengths(self) -> tuple[tuple[int, ...], ...]:
        audio_latents = [entry.audio_latent for entry in self.memory]
        if not audio_latents or any(item is None for item in audio_latents):
            return ()
        return (
            tuple(
                int(audio_latent.shape[1])
                for audio_latent in audio_latents
                if audio_latent is not None
            ),
        )

    def __len__(self) -> int:
        return len(self.memory)

View on GitHub (pinned to 0132848349)

Solutions

1. Re-encode all memory audio with the same codec/config so latents share B and C dims; rebuild the memory bank
2. Audit each saved entry's audio_latent.shape and evict/re-save the outliers
3. Ensure every save path goes through _prepare_audio_latent and enforces B==1

Example fix

# before
bank.save_memory_slot("a", audio_latent=lat_64ch)   # [1, T, 64]
bank.save_memory_slot("b", audio_latent=lat_128ch) # [1, T, 128]
merged = bank.get_memory_audio()
# after
bank.save_memory_slot("a", audio_latent=encode(audio))  # same codec for all
bank.save_memory_slot("b", audio_latent=encode(audio))
merged = bank.get_memory_audio()

shapes = {(e.audio_latent.shape[0], e.audio_latent.shape[2]) for e in bank.memory if e.audio_latent is not None}
if len(shapes) > 1:
    raise ValueError(f"memory bank has inconsistent audio latent dims: {shapes}; re-encode all slots with one codec")

def bank_audio_is_consistent(bank) -> bool:
    shapes = {(e.audio_latent.shape[0], e.audio_latent.shape[2]) for e in bank.memory if e.audio_latent is not None}
    return len(shapes) <= 1

When it happens

Trigger: Calling get_memory_audio after saving slots with latents from different audio codecs or channel widths (e.g. one [1, T, 64] and another [1, T, 128]), or after saving one latent with batch 1 and another with batch 2.

Common situations: Mixing latents from different model versions/checkpoints in one memory bank; saving a multi-batch latent via a path that bypassed validation; changing audio codec config between saves in a long-running session.

Related errors

• Expected audio_latent shape [B, T, C], got shape={tuple(audi
• Expected batch size 1 for decoded audio, got shape={tuple(wa
• Expected decoded audio with 1, 2, or 3 dims, got shape={tupl
• paired audio memory slot requires audio_latent
• audio_out_channels must be divisible by tp_size for TP-shard

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0a3b9855ac2cebbb. Report an issue: GitHub.