vllm-project/vllm · critical · TimeoutError

Timed out waiting for EC mmap file to reach {expected_size}

Error message

Timed out waiting for EC mmap file to reach {expected_size} bytes

What it means

ECSharedRegion maps a file that one process (the creator) truncates to the full expected size while other TP workers mmap it. _wait_for_file_size spin-waits (5 ms interval, default 30 s timeout) for os.fstat(fd).st_size to reach expected_size; if the creator never completes ftruncate — because it crashed, is stuck, or filesystem latency exceeds the window — TimeoutError is raised.

Source

Thrown at vllm/distributed/ec_transfer/ec_connector/cpu/ec_shared_region.py:28

import mmap
import os
import time

import torch

from vllm.logger import init_logger

logger = init_logger(__name__)


def _wait_for_file_size(fd: int, expected_size: int, timeout: float = 30.0):
    """Spin-wait until the file reaches expected_size (creator truncated it)."""
    deadline = time.monotonic() + timeout
    while True:
        if os.fstat(fd).st_size >= expected_size:
            return
        if time.monotonic() > deadline:
            raise TimeoutError(
                f"Timed out waiting for EC mmap file to reach {expected_size} bytes"
            )
        time.sleep(0.005)


class ECSharedRegion:
    """Flat mmap-backed memory region shared across TP workers for
    encoder cache blocks.

    Layout: (num_blocks, block_size_bytes) — contiguous, no per-worker
    interleaving. All workers map the same file and see identical data.

    File path: /dev/shm/vllm_ec_{engine_id}.mmap

    This class owns only the shared memory substrate (mmap lifecycle, the
    `blocks` view, CUDA host registration). Block allocation and eviction
    are tracked by `EmbeddingCache` in the scheduler process.
    """

View on GitHub (pinned to c794754062)

Solutions

1. Check whether the creator process is alive and its logs for a crash during ftruncate (raise the creator's error first)
2. Ensure all processes compute the same expected_size: same model dtype, hidden dim, and ec_cpu_bytes (shared region sizing must be identical)
3. If on slow shared storage, place the mmap file on local tmpfs/disk or raise the timeout parameter
4. Remove half-created files from a previous crashed run before restarting

# before mapping, confirm the creator is alive and size will converge
size = os.fstat(fd).st_size
if size < expected_size:
    assert creator_process.is_alive(), "creator died before ftruncate"

try:
    _wait_for_file_size(fd, expected_size, timeout=30.0)
except TimeoutError:
    if not creator_alive():
        raise RuntimeError("EC region creator crashed") from None
    _wait_for_file_size(fd, expected_size, timeout=120.0)  # slow-fs retry

When it happens

Trigger: The creating process dying between file creation and ftruncate (OOM kill, startup error); the mmap file living on a slow network filesystem (NFS) where metadata propagation lags; a size mismatch where the creator truncated to a smaller size (e.g. num_blocks computed differently due to a dtype/hidden-size mismatch between processes).

Common situations: Multi-TP encoder-cache transfer startup where one rank fails during init and others block on the file; /tmp mapped onto unusual storage in containers; heterogeneous config where producer and consumer compute different expected_size from different model configs.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• startup handshake timed out while waiting for {stage} after
• engine input registration timed out after {timeout:?}
• Tokenizer error: {0}
• cannot use in-process coordinator with bootstrapped transpor
• invalid --allowed-methods value {method:?}: {e}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/a13b2211beea53b7. Report an issue: GitHub.