sgl-project/sglang · error · ValueError

Neighborhood attention requires each dim to be at least its

Error message

Neighborhood attention requires each dim to be at least its kernel size; got (T, H, W) = ({num_frames}, {height}, {width}) with kernel_size {self.kernel_size}.

What it means

Neighborhood (windowed) attention needs every spatial/temporal dimension to be at least as large as its attention kernel so each position has a full neighborhood. The block checks (T,H,W) of the incoming hidden_states against kernel_size at forward time and raises when any dim is smaller than its kernel extent.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/decoders/ltx_2_5_diffusion_decoder.py:424

        return query, key, value

    def build_block_mask(self, hidden_states: torch.Tensor):
        """The window mask for this grid, or `None` when NATTEN handles it.

        Fixed within a stage, so built once.
        """
        if _na3d() is not None:
            return None
        num_frames, height, width = hidden_states.shape[1:4]
        return _neighborhood_block_mask(
            num_frames, height, width, self.kernel_size, hidden_states.device
        )

    def forward(self, hidden_states: torch.Tensor, block_mask=None) -> torch.Tensor:
        batch_size, num_frames, height, width, _ = hidden_states.shape
        kernel_t, kernel_h, kernel_w = self.kernel_size
        if num_frames < kernel_t or height < kernel_h or width < kernel_w:
            raise ValueError(
                "Neighborhood attention requires each dim to be at least its "
                f"kernel size; got (T, H, W) = ({num_frames}, {height}, {width}) "
                f"with kernel_size {self.kernel_size}."
            )

        query, key, value = self.project_qkv(hidden_states)

        na3d = _na3d()
        if na3d is not None:
            # `project_qkv` already yields NATTEN's layout. scale=1.0: the
            # query is pre-scaled there.
            hidden_states = na3d(
                query, key, value, kernel_size=self.kernel_size, scale=1.0
            )
            hidden_states = hidden_states.reshape(
                batch_size, num_frames, height, width, self.heads * self.head_dim
            )
            return self.to_out[0](hidden_states)

View on GitHub (pinned to 0132848349)

Solutions

1. Increase resolution/frame count so T>=kernel_t, H>=kernel_h, W>=kernel_w (e.g. at least 3 frames and 3x3 latent grid for kernel (3,3,3))
2. If small inputs must be supported, pad the latent grid and crop after decoding
3. Use a decoder config with smaller kernel_size for small-input workloads
4. Validate requested resolution/frames against the decoder kernel before running the pipeline

Example fix

# before
out = block(torch.randn(1, 1, 2, 2, C))  # T=1 < kernel_t=3
# after
out = block(torch.randn(1, 3, 4, 4, C))  # all dims >= kernel extents

kt, kh, kw = block.kernel_size
T, H, W = num_frames, height, width
if T < kt or H < kh or W < kw:
    raise ValueError(f"input (T,H,W)=({T},{H},{W}) smaller than kernel {block.kernel_size}; "
                     "increase resolution/frames or pad")

def fits_kernel(t: int, h: int, w: int, kernel: tuple[int,int,int]) -> bool:
    kt, kh, kw = kernel
    return t >= kt and h >= kh and w >= kw

try:
    out = block(hidden_states)
except ValueError as e:
    if "kernel size" in str(e):
        pad = (max(0,kt-T), max(0,kh-H), max(0,kw-W))  # F.pad latent then crop output
        raise
    raise

When it happens

Trigger: Calling block.forward(hidden_states) where hidden_states has shape (B, T, H, W, C) with, e.g., T < kernel_t (fewer frames than the temporal kernel), or H/W smaller than the spatial kernel — common with tiny test videos, thumbnails, or heavily downsampled latents.

Common situations: Generating a 1-frame or very short clip with a (3,3,3)-kernel decoder; small resolutions like 32x32 latents after patching; unit tests using minimal dummy tensors; user requests for tiny aspect ratios.

Related errors

• Invalid {self.vae_scale_factor=}. Must be > 0.
• Invalid {self.patch_size=}. Must be > 0.
• Invalid latent H/W computed from batch.height/width: {batch.
• Invalid spatial patching for packed token latents. Expected
• Invalid tokens_per_frame={tokens_per_frame} from {latent_hei

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/79184a9cf0de1fbf. Report an issue: GitHub.