{"record":{"id":"f4dace8e8ce4ddee","repo":"invoke-ai/InvokeAI","slug":"image-to-prompt-generation-stalled-no-output-for","errorCode":null,"errorMessage":"Image-to-prompt generation stalled (no output for {STREAM_TIMEOUT}s)","messagePattern":"Image-to-prompt generation stalled \\(no output for (.+?)s\\)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/llava_onevision_pipeline.py","lineNumber":108,"sourceCode":"            if progress_callback is not None:\n                progress_callback(min(token_count, max_new_tokens), max_new_tokens)\n\n        try:\n            for chunk in streamer:\n                if not chunk:\n                    continue\n                chunks.append(chunk)\n                now = time.monotonic()\n                if progress_callback is not None and now - last_emit >= PROGRESS_EMIT_INTERVAL:\n                    _emit_progress()\n                    last_emit = now\n        except queue.Empty as e:\n            # The streamer timed out waiting for the next token: generate() stalled\n            # without raising and without signalling end(). Surface any captured error,\n            # otherwise raise a timeout rather than block on thread.join() below.\n            if generation_error:\n                raise generation_error[0] from e\n            raise RuntimeError(f\"Image-to-prompt generation stalled (no output for {STREAM_TIMEOUT}s)\") from e\n\n        # Guarantee a final emission so the reported token count is exact even if the\n        # last increment was throttled.\n        if progress_callback is not None and chunks:\n            _emit_progress()\n\n        thread.join()\n        if generation_error:\n            raise generation_error[0]\n\n        return \"\".join(chunks).strip()\n","sourceCodeStart":90,"sourceCodeEnd":120,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/llava_onevision_pipeline.py#L90-L120","documentation":"run() generates tokens on a worker thread and streams them through a queue with a STREAM_TIMEOUT-second wait. If queue.Empty fires and no worker exception was captured, generate() stalled without output and without signaling end(); the code surfaces any captured error first, otherwise raises this RuntimeError instead of blocking forever on thread.join().","triggerScenarios":"TextStreamer emitting nothing for STREAM_TIMEOUT seconds during LLaVA OneVision inference — model hang on GPU/OOM, deadlock in the generation thread, extremely long prefill, or CUDA driver stall.","commonSituations":"GPU memory exhaustion leaving generate() wedged; a slow/oversubscribed GPU where prefill exceeds the timeout; driver/library bugs; very long chat templates with many large images.","solutions":["Retry generation once; transient stalls (slow prefill, contention) often succeed on a second attempt.","Reduce input size: fewer/smaller images and shorter prompts to cut prefill time.","Check GPU memory and logs for OOM/CUDA errors; free VRAM or lower resolution/dtype.","Increase STREAM_TIMEOUT if your hardware legitimately takes longer than the limit.","Upgrade transformers/torch — some streamer/generation deadlocks are fixed upstream."],"exampleFix":"// before\nprompt = pipeline.run(images=imgs, max_new_tokens=400)  # stalls on slow GPU\n// after\ntry:\n    prompt = pipeline.run(images=imgs, max_new_tokens=400)\nexcept RuntimeError as e:\n    if 'stalled' in str(e):\n        prompt = pipeline.run(images=[resize_smaller(i) for i in imgs], max_new_tokens=400)","handlingStrategy":"retry","validationCode":"# Pre-check: ensure a CUDA device is available and not near OOM before long generation\nfree, total = torch.cuda.mem_get_info()\nif free / total < 0.1:\n    torch.cuda.empty_cache()  # avoid stalls caused by memory pressure","typeGuard":null,"tryCatchPattern":"for attempt in range(2):\n    try:\n        prompt = pipeline.run(images, dtype=dtype, max_new_tokens=400, progress_callback=cb)\n        break\n    except RuntimeError as e:\n        if 'generation stalled' in str(e) and attempt == 0:\n            torch.cuda.empty_cache()\n            continue\n        raise","preventionTips":["Reduce image count/size and prompt length to keep prefill within timeout","Monitor VRAM; free memory before long LLaVA generations","Catch queue.Empty-rooted RuntimeErrors and retry once with a smaller input","Keep transformers/torch updated for streamer deadlock fixes","Raise STREAM_TIMEOUT on slow hardware rather than disabling the guard"],"tags":["llava-onevision","timeout","inference-stall","streaming"],"backgroundTag":"inference-generation-timeout","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}