sgl-project/sglang · error · ValueError

JoyImage conditioning batch mismatch: hidden_states batch={b

Error message

JoyImage conditioning batch mismatch: hidden_states batch={batch_size}, encoder_hidden_states batch={cond_batch}.

What it means

JoyImage allows the conditioning (encoder_hidden_states) batch to be smaller than the latent batch only when the latent batch is an exact integer multiple (e.g. N CFG samples per condition). If the conditioning batch is not a positive divisor of the latent batch, shapes cannot be reconciled and forward aborts.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/joy_image.py:497

        forward_batch = get_forward_context().forward_batch
        sequence_shard_enabled = (
            forward_batch is not None
            and getattr(forward_batch, "enable_sequence_shard", False)
            and self.sp_size > 1
        )

        batch_size = hidden_states.shape[0]

        if not isinstance(encoder_hidden_states, torch.Tensor):
            encoder_hidden_states = encoder_hidden_states[0]

        if isinstance(encoder_hidden_states_mask, list):
            encoder_hidden_states_mask = encoder_hidden_states_mask[0]

        cond_batch = int(encoder_hidden_states.shape[0])
        if cond_batch != int(batch_size):
            if cond_batch <= 0 or int(batch_size) % cond_batch != 0:
                raise ValueError(
                    "JoyImage conditioning batch mismatch: "
                    f"hidden_states batch={batch_size}, "
                    f"encoder_hidden_states batch={cond_batch}."
                )
            repeat_factor = int(batch_size) // cond_batch
            encoder_hidden_states = encoder_hidden_states.repeat_interleave(
                repeat_factor, dim=0
            )
            if encoder_hidden_states_mask is not None:
                encoder_hidden_states_mask = (
                    encoder_hidden_states_mask.repeat_interleave(repeat_factor, dim=0)
                )

        # Prepare img
        x = rearrange(hidden_states, "b n c p1 p2 p3 -> (b n) c p1 p2 p3")
        x = self.img_in(x)
        img = rearrange(x, "(b n) d 1 1 1 -> b n d", b=batch_size)

View on GitHub (pinned to 0132848349)

Solutions

1. Make the latent batch an exact multiple of the conditioning batch (typically B = 2*C for CFG, or B = C without CFG)
2. repeat_interleave the conditioning yourself to exactly B rows before calling forward
3. Check the scheduler/batcher that produced hidden_states for off-by-one guidance duplication

Example fix

// before
noise = torch.randn(3, ...)          # batch 3
cond = encode(prompts)              # batch 2
dit(noise, encoder_hidden_states=cond)

// after
noise = torch.randn(4, ...)          # 2 * cond batch for CFG
cond = encode(prompts)               # batch 2
dit(noise, encoder_hidden_states=cond)

B = hidden_states.shape[0]; C = encoder_hidden_states.shape[0]\nassert C > 0 and B % C == 0, f'latent batch {B} must be a positive multiple of cond batch {C}'

def conditioning_batches_align(h: torch.Tensor, c: torch.Tensor) -> bool:\n    return c.shape[0] > 0 and h.shape[0] % c.shape[0] == 0

When it happens

Trigger: hidden_states has batch B but encoder_hidden_states has batch C where B % C != 0 or C <= 0, e.g. latents batch=3 (odd CFG/batched mix) with a single conditioning tensor of batch=2.

Common situations: Batching requests with different guidance scales so B is not 2*C; per-prompt conditioning counts not aligned with the latent batch; empty conditioning tensor (batch 0).

Related errors

• Fused QK-Norm + RoPE kernel only supports float16/bfloat16,
• txt_freqs_cis must be a 2D cos_sin_cache tensor
• The number of initial states is expected to be equal to the
• Mismatched batch sizes: mixed_qkv.shape[0]={B}, a.shape[0]={
• Mismatched batch sizes: mixed_qkv.shape[0]={B}, a.shape[0]={

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/713fddedc2729b02. Report an issue: GitHub.