sgl-project/sglang · error · TypeError

CUDA VMM feature transport requires each feature field to co

Error message

CUDA VMM feature transport requires each feature field to contain a single tensor

What it means

wrap_items validates that each feature field ('feature', 'precomputed_embeddings') is a single torch.Tensor or non-tensor value — never a list/dict/nested structure containing tensors. Packed VMM transport maps each field to one contiguous pool slice, so heterogeneous tensor containers are unsupported and fail closed.

Source

Thrown at python/sglang/srt/utils/cuda_vmm_transport_utils.py:1010

            if len(pack_candidates) >= 2:
                packed = self.pool.wrap_tensors(
                    [tensor for _, tensor in pack_candidates]
                )
                if packed is not None:
                    for (item, tensor), proxy in zip(
                        pack_candidates, packed, strict=True
                    ):
                        item.feature = proxy
                        updates.append((item, "feature", tensor, proxy))

            for item in mm_items:
                fields = (
                    ("feature", item.feature),
                    ("precomputed_embeddings", item.precomputed_embeddings),
                )
                for field, tensor in fields:
                    if _contains_tensor_container(tensor):
                        raise TypeError(
                            "CUDA VMM feature transport requires each feature "
                            "field to contain a single tensor"
                        )
                    if not isinstance(tensor, torch.Tensor):
                        continue
                    wrapped = self.pool.wrap_tensor(tensor)
                    setattr(item, field, wrapped)
                    updates.append((item, field, tensor, wrapped))
        except BaseException as error:
            rollback_errors = []
            for item, field, tensor, wrapped in reversed(updates):
                try:
                    if isinstance(wrapped, CudaVmmTensorTransportProxy):
                        self.pool.cancel_proxy(wrapped)
                except BaseException as rollback_error:
                    rollback_errors.append(rollback_error)
                finally:
                    setattr(item, field, tensor)

View on GitHub (pinned to 0132848349)

Solutions

1. Flatten/collapse multi-tensor fields into a single tensor (cat/stack) in the model's mm processor before dispatch
2. Disable cuda_vmm transport for models with container-shaped features
3. Add a pre-dispatch validation step that rejects container fields early with a clear message

Example fix

# before
item.feature = [feat_frame0, feat_frame1]

# after
item.feature = torch.cat([feat_frame0, feat_frame1], dim=0)

def is_flat_tensor(v) -> bool:
    return v is None or isinstance(v, torch.Tensor)

def is_wrap_safe(item) -> bool:
    return all(
        _contains_tensor_container(v) is False
        for v in (item.feature, item.precomputed_embeddings)
    )

try:
    transport.wrap_items(items)
except TypeError as e:
    if "single tensor" in str(e):
        items = flatten_feature_tensors(items)
        transport.wrap_items(items)

When it happens

Trigger: A model returning feature=[t1, t2] or a dict of tensors in MultiModalItem fields; precomputed_embeddings as a nested list; new model integration with multi-tensor features fed into cuda_vmm transport.

Common situations: Integrating a new multimodal model whose processor emits per-frame/per-crop tensor lists; switching transport from default (which tolerates containers) to cuda_vmm.

Related errors

• handle_types must be 'auto', an integer, or None
• memory_size must be positive
• consumer_count must be positive
• recycle_interval must be positive
• consumer_count must be 1, the attention TP size, or the full

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1c74d9d6c0943168. Report an issue: GitHub.