RVC-Boss/GPT-SoVITS · error · RuntimeError

CUDA Graph T2S inference failed

Error message

CUDA Graph T2S inference failed

What it means

RuntimeError raised after a CUDA-Graph-accelerated T2S (GPT semantic stage) generate() call returns a non-None t2s_result.exception. CUDA Graphs replay a captured fixed-shape kernel sequence; any input whose shape/padding exceeds the captured graph (or a device-side fault during replay) makes the wrapper return the captured exception instead of raising inside, so the caller re-raises it as this generic error after printing the original traceback.

Source

Thrown at GPT_SoVITS/inference_webui.py:943

                from AR.models.structs_cudagraph import T2SRequest
                with torch.no_grad():
                    t2s_request = T2SRequest(
                        [all_phoneme_ids.squeeze(0)],
                        all_phoneme_len,
                        all_phoneme_ids.new_zeros((1, 0)) if ref_free else prompt,
                        [bert.squeeze(0)],
                        valid_length=1,
                        top_k=top_k,
                        top_p=top_p,
                        temperature=temperature,
                        early_stop_num=hz * max_sec,
                        use_cuda_graph=True,
                    )
                    t2s_result = t2s_model_cudagraph.generate(t2s_request)
                    if t2s_result.exception is not None:
                        print(t2s_result.exception)
                        print(t2s_result.traceback)
                        raise RuntimeError("CUDA Graph T2S inference failed")
                    pred_semantic = t2s_result.result[0].unsqueeze(0).unsqueeze(0)
                    cache[i_text] = pred_semantic
            else:
                with torch.no_grad():
                    pred_semantic, idx = t2s_model.model.infer_panel(
                        all_phoneme_ids,
                        all_phoneme_len,
                        None if ref_free else prompt,
                        bert,
                        # prompt_phone_len=ph_offset,
                        top_k=top_k,
                        top_p=top_p,
                        temperature=temperature,
                        early_stop_num=hz * max_sec,
                    )
                    pred_semantic = pred_semantic[:, -idx:].unsqueeze(0)
                    cache[i_text] = pred_semantic
        t3 = ttime()

View on GitHub (pinned to d523079fc0)

Solutions

1. Look at the two lines printed immediately before the raise (t2s_result.exception and traceback) — they contain the real root cause; fix that first.
2. Shorten/split the input text so semantic length stays within the graph's captured length (early_stop_num = hz * max_sec).
3. Disable the CUDA graph path (t2s_model_cudagraph = None / use the non-cudagraph infer_panel branch) to confirm whether it is graph-specific; keep it off if unstable on your GPU.
4. Update CUDA/driver and PyTorch to a consistent pair; re-warm/rebuild the graph after any model or precision change.

Example fix

# before
t2s_result = t2s_model_cudagraph.generate(t2s_request)
if t2s_result.exception is not None:
    raise RuntimeError("CUDA Graph T2S inference failed")

# after: fall back to standard inference on graph failure
if t2s_result.exception is not None:
    print(t2s_result.exception, t2s_result.traceback)
    with torch.no_grad():
        pred_semantic, idx = t2s_model.model.infer_panel(
            all_phoneme_ids, all_phoneme_len, prompt, bert,
            top_k=top_k, top_p=top_p, temperature=temperature,
            early_stop_num=hz * max_sec,
        )

# bound input length to the graph's budget before generate
max_semantic = hz * max_sec
if estimated_phoneme_len(text) * hz > max_semantic:
    text = split_into_sentences(text)[0]  # or process sentence-by-sentence

t2s_result = t2s_model_cudagraph.generate(t2s_request)
if t2s_result.exception is not None:
    print(t2s_result.exception, t2s_result.traceback)   # real cause
    pred_semantic, idx = t2s_model.model.infer_panel(     # fallback path
        all_phoneme_ids, all_phoneme_len, prompt, bert,
        top_k=top_k, top_p=top_p, temperature=temperature,
        early_stop_num=hz * max_sec,
    )

When it happens

Trigger: In inference_webui.py, when a cached text (i_text) is reused and t2s_model_cudagraph.generate(t2s_request) is called with use_cuda_graph=True and the request exceeds graph limits (sequence length beyond captured max, batch shape mismatch) or hits a CUDA error; the true cause is printed via t2s_result.exception/traceback just above the raise.

Common situations: Very long input text exceeding the captured max_sec/graph length budget; concurrent requests racing the shared cudagraph instance; CUDA driver/JIT mismatch after upgrade; half-precision overflow on some GPUs; memory pressure corrupting capture.

AI-assisted analysis of RVC-Boss/GPT-SoVITS@d523079fc0 (2026-08-15). Data as JSON: /api/errors/9e09bca2a91684df. Report an issue: GitHub.