invoke-ai/InvokeAI · error · RuntimeError

Image-to-prompt generation stalled (no output for {STREAM_TI

Error message

Image-to-prompt generation stalled (no output for {STREAM_TIMEOUT}s)

What it means

run() generates tokens on a worker thread and streams them through a queue with a STREAM_TIMEOUT-second wait. If queue.Empty fires and no worker exception was captured, generate() stalled without output and without signaling end(); the code surfaces any captured error first, otherwise raises this RuntimeError instead of blocking forever on thread.join().

Source

Thrown at invokeai/backend/llava_onevision_pipeline.py:108

            if progress_callback is not None:
                progress_callback(min(token_count, max_new_tokens), max_new_tokens)

        try:
            for chunk in streamer:
                if not chunk:
                    continue
                chunks.append(chunk)
                now = time.monotonic()
                if progress_callback is not None and now - last_emit >= PROGRESS_EMIT_INTERVAL:
                    _emit_progress()
                    last_emit = now
        except queue.Empty as e:
            # The streamer timed out waiting for the next token: generate() stalled
            # without raising and without signalling end(). Surface any captured error,
            # otherwise raise a timeout rather than block on thread.join() below.
            if generation_error:
                raise generation_error[0] from e
            raise RuntimeError(f"Image-to-prompt generation stalled (no output for {STREAM_TIMEOUT}s)") from e

        # Guarantee a final emission so the reported token count is exact even if the
        # last increment was throttled.
        if progress_callback is not None and chunks:
            _emit_progress()

        thread.join()
        if generation_error:
            raise generation_error[0]

        return "".join(chunks).strip()

View on GitHub (pinned to 0b6a024f2f)

Solutions

Retry generation once; transient stalls (slow prefill, contention) often succeed on a second attempt.
Reduce input size: fewer/smaller images and shorter prompts to cut prefill time.
Check GPU memory and logs for OOM/CUDA errors; free VRAM or lower resolution/dtype.
Increase STREAM_TIMEOUT if your hardware legitimately takes longer than the limit.
Upgrade transformers/torch — some streamer/generation deadlocks are fixed upstream.

Example fix

// before
prompt = pipeline.run(images=imgs, max_new_tokens=400)  # stalls on slow GPU
// after
try:
    prompt = pipeline.run(images=imgs, max_new_tokens=400)
except RuntimeError as e:
    if 'stalled' in str(e):
        prompt = pipeline.run(images=[resize_smaller(i) for i in imgs], max_new_tokens=400)

Defensive patterns

Strategy: retry

Validate before calling

# Pre-check: ensure a CUDA device is available and not near OOM before long generation
free, total = torch.cuda.mem_get_info()
if free / total < 0.1:
    torch.cuda.empty_cache()  # avoid stalls caused by memory pressure

Try / catch

for attempt in range(2):
    try:
        prompt = pipeline.run(images, dtype=dtype, max_new_tokens=400, progress_callback=cb)
        break
    except RuntimeError as e:
        if 'generation stalled' in str(e) and attempt == 0:
            torch.cuda.empty_cache()
            continue
        raise

Prevention

Reduce image count/size and prompt length to keep prefill within timeout
Monitor VRAM; free memory before long LLaVA generations
Catch queue.Empty-rooted RuntimeErrors and retry once with a smaller input
Keep transformers/torch updated for streamer deadlock fixes
Raise STREAM_TIMEOUT on slow hardware rather than disabling the guard

When it happens

Trigger: TextStreamer emitting nothing for STREAM_TIMEOUT seconds during LLaVA OneVision inference — model hang on GPU/OOM, deadlock in the generation thread, extremely long prefill, or CUDA driver stall.

Common situations: GPU memory exhaustion leaving generate() wedged; a slow/oversubscribed GPU where prefill exceeds the timeout; driver/library bugs; very long chat templates with many large images.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/f4dace8e8ce4ddee. Report an issue: GitHub.