{"record":{"id":"a13b2211beea53b7","repo":"vllm-project/vllm","slug":"timed-out-waiting-for-ec-mmap-file-to-reach-expec","errorCode":null,"errorMessage":"Timed out waiting for EC mmap file to reach {expected_size} bytes","messagePattern":"Timed out waiting for EC mmap file to reach (.+?) bytes","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/ec_transfer/ec_connector/cpu/ec_shared_region.py","lineNumber":28,"sourceCode":"import mmap\nimport os\nimport time\n\nimport torch\n\nfrom vllm.logger import init_logger\n\nlogger = init_logger(__name__)\n\n\ndef _wait_for_file_size(fd: int, expected_size: int, timeout: float = 30.0):\n    \"\"\"Spin-wait until the file reaches expected_size (creator truncated it).\"\"\"\n    deadline = time.monotonic() + timeout\n    while True:\n        if os.fstat(fd).st_size >= expected_size:\n            return\n        if time.monotonic() > deadline:\n            raise TimeoutError(\n                f\"Timed out waiting for EC mmap file to reach {expected_size} bytes\"\n            )\n        time.sleep(0.005)\n\n\nclass ECSharedRegion:\n    \"\"\"Flat mmap-backed memory region shared across TP workers for\n    encoder cache blocks.\n\n    Layout: (num_blocks, block_size_bytes) — contiguous, no per-worker\n    interleaving. All workers map the same file and see identical data.\n\n    File path: /dev/shm/vllm_ec_{engine_id}.mmap\n\n    This class owns only the shared memory substrate (mmap lifecycle, the\n    `blocks` view, CUDA host registration). Block allocation and eviction\n    are tracked by `EmbeddingCache` in the scheduler process.\n    \"\"\"","sourceCodeStart":10,"sourceCodeEnd":46,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/ec_transfer/ec_connector/cpu/ec_shared_region.py#L10-L46","documentation":"ECSharedRegion maps a file that one process (the creator) truncates to the full expected size while other TP workers mmap it. _wait_for_file_size spin-waits (5 ms interval, default 30 s timeout) for os.fstat(fd).st_size to reach expected_size; if the creator never completes ftruncate — because it crashed, is stuck, or filesystem latency exceeds the window — TimeoutError is raised.","triggerScenarios":"The creating process dying between file creation and ftruncate (OOM kill, startup error); the mmap file living on a slow network filesystem (NFS) where metadata propagation lags; a size mismatch where the creator truncated to a smaller size (e.g. num_blocks computed differently due to a dtype/hidden-size mismatch between processes).","commonSituations":"Multi-TP encoder-cache transfer startup where one rank fails during init and others block on the file; /tmp mapped onto unusual storage in containers; heterogeneous config where producer and consumer compute different expected_size from different model configs.","solutions":["Check whether the creator process is alive and its logs for a crash during ftruncate (raise the creator's error first)","Ensure all processes compute the same expected_size: same model dtype, hidden dim, and ec_cpu_bytes (shared region sizing must be identical)","If on slow shared storage, place the mmap file on local tmpfs/disk or raise the timeout parameter","Remove half-created files from a previous crashed run before restarting"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"# before mapping, confirm the creator is alive and size will converge\nsize = os.fstat(fd).st_size\nif size < expected_size:\n    assert creator_process.is_alive(), \"creator died before ftruncate\"","typeGuard":null,"tryCatchPattern":"try:\n    _wait_for_file_size(fd, expected_size, timeout=30.0)\nexcept TimeoutError:\n    if not creator_alive():\n        raise RuntimeError(\"EC region creator crashed\") from None\n    _wait_for_file_size(fd, expected_size, timeout=120.0)  # slow-fs retry","preventionTips":["Surface the creator process's error before waiting peers time out","Keep all ranks' model dtype/hidden_size/ec_cpu_bytes identical so expected_size matches","Place the mmap file on local fast storage, not NFS","Clean up partial files from crashed runs before restart"],"tags":["ec-transfer","timeout","filesystem","startup"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}