{"record":{"id":"9e09bca2a91684df","repo":"RVC-Boss/GPT-SoVITS","slug":"cuda-graph-t2s-inference-failed","errorCode":null,"errorMessage":"CUDA Graph T2S inference failed","messagePattern":"CUDA Graph T2S inference failed","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"GPT_SoVITS/inference_webui.py","lineNumber":943,"sourceCode":"                from AR.models.structs_cudagraph import T2SRequest\r\n                with torch.no_grad():\r\n                    t2s_request = T2SRequest(\r\n                        [all_phoneme_ids.squeeze(0)],\r\n                        all_phoneme_len,\r\n                        all_phoneme_ids.new_zeros((1, 0)) if ref_free else prompt,\r\n                        [bert.squeeze(0)],\r\n                        valid_length=1,\r\n                        top_k=top_k,\r\n                        top_p=top_p,\r\n                        temperature=temperature,\r\n                        early_stop_num=hz * max_sec,\r\n                        use_cuda_graph=True,\r\n                    )\r\n                    t2s_result = t2s_model_cudagraph.generate(t2s_request)\r\n                    if t2s_result.exception is not None:\r\n                        print(t2s_result.exception)\r\n                        print(t2s_result.traceback)\r\n                        raise RuntimeError(\"CUDA Graph T2S inference failed\")\r\n                    pred_semantic = t2s_result.result[0].unsqueeze(0).unsqueeze(0)\r\n                    cache[i_text] = pred_semantic\r\n            else:\r\n                with torch.no_grad():\r\n                    pred_semantic, idx = t2s_model.model.infer_panel(\r\n                        all_phoneme_ids,\r\n                        all_phoneme_len,\r\n                        None if ref_free else prompt,\r\n                        bert,\r\n                        # prompt_phone_len=ph_offset,\r\n                        top_k=top_k,\r\n                        top_p=top_p,\r\n                        temperature=temperature,\r\n                        early_stop_num=hz * max_sec,\r\n                    )\r\n                    pred_semantic = pred_semantic[:, -idx:].unsqueeze(0)\r\n                    cache[i_text] = pred_semantic\r\n        t3 = ttime()\r","sourceCodeStart":925,"sourceCodeEnd":961,"githubUrl":"https://github.com/RVC-Boss/GPT-SoVITS/blob/d523079fc05d9a8028d6085bffe4a2757c32abb6/GPT_SoVITS/inference_webui.py#L925-L961","documentation":"RuntimeError raised after a CUDA-Graph-accelerated T2S (GPT semantic stage) generate() call returns a non-None t2s_result.exception. CUDA Graphs replay a captured fixed-shape kernel sequence; any input whose shape/padding exceeds the captured graph (or a device-side fault during replay) makes the wrapper return the captured exception instead of raising inside, so the caller re-raises it as this generic error after printing the original traceback.","triggerScenarios":"In inference_webui.py, when a cached text (i_text) is reused and t2s_model_cudagraph.generate(t2s_request) is called with use_cuda_graph=True and the request exceeds graph limits (sequence length beyond captured max, batch shape mismatch) or hits a CUDA error; the true cause is printed via t2s_result.exception/traceback just above the raise.","commonSituations":"Very long input text exceeding the captured max_sec/graph length budget; concurrent requests racing the shared cudagraph instance; CUDA driver/JIT mismatch after upgrade; half-precision overflow on some GPUs; memory pressure corrupting capture.","solutions":["Look at the two lines printed immediately before the raise (t2s_result.exception and traceback) — they contain the real root cause; fix that first.","Shorten/split the input text so semantic length stays within the graph's captured length (early_stop_num = hz * max_sec).","Disable the CUDA graph path (t2s_model_cudagraph = None / use the non-cudagraph infer_panel branch) to confirm whether it is graph-specific; keep it off if unstable on your GPU.","Update CUDA/driver and PyTorch to a consistent pair; re-warm/rebuild the graph after any model or precision change."],"exampleFix":"# before\nt2s_result = t2s_model_cudagraph.generate(t2s_request)\nif t2s_result.exception is not None:\n    raise RuntimeError(\"CUDA Graph T2S inference failed\")\n\n# after: fall back to standard inference on graph failure\nif t2s_result.exception is not None:\n    print(t2s_result.exception, t2s_result.traceback)\n    with torch.no_grad():\n        pred_semantic, idx = t2s_model.model.infer_panel(\n            all_phoneme_ids, all_phoneme_len, prompt, bert,\n            top_k=top_k, top_p=top_p, temperature=temperature,\n            early_stop_num=hz * max_sec,\n        )","handlingStrategy":"fallback","validationCode":"# bound input length to the graph's budget before generate\nmax_semantic = hz * max_sec\nif estimated_phoneme_len(text) * hz > max_semantic:\n    text = split_into_sentences(text)[0]  # or process sentence-by-sentence","typeGuard":null,"tryCatchPattern":"t2s_result = t2s_model_cudagraph.generate(t2s_request)\nif t2s_result.exception is not None:\n    print(t2s_result.exception, t2s_result.traceback)   # real cause\n    pred_semantic, idx = t2s_model.model.infer_panel(     # fallback path\n        all_phoneme_ids, all_phoneme_len, prompt, bert,\n        top_k=top_k, top_p=top_p, temperature=temperature,\n        early_stop_num=hz * max_sec,\n    )","preventionTips":["Always log t2s_result.exception/traceback — the RuntimeError itself carries no cause.","Keep input text within the graph's captured max length; split long text first.","Rebuild the CUDA graph (t2s_model_cudagraph = None) after model/precision changes; keep a non-graph fallback enabled.","Keep CUDA driver, PyTorch, and the GPU arch consistent; avoid sharing one graph instance across threads."],"tags":["cuda","cuda-graph","t2s","gpu","inference"],"backgroundTag":null,"analyzedSha":"d523079fc05d9a8028d6085bffe4a2757c32abb6","analyzedAt":"2026-08-15T01:06:46.402Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}