{"record":{"id":"061a349d979bdb82","repo":"sgl-project/sglang","slug":"role-role-type-value-rank-rank-idx-failed-to-i","errorCode":null,"errorMessage":"Role {role_type.value} rank {rank_idx} failed to initialize.","messagePattern":"Role (.+?) rank (.+?) failed to initialize\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/launch_server.py","lineNumber":774,"sourceCode":"            daemon=True,\n        )\n        process.start()\n        processes.append(process)\n        readers.append(reader)\n\n    # Wait for all ranks to be ready (after all are spawned)\n    for rank_idx, reader in enumerate(readers):\n        try:\n            data = reader.recv()\n        except EOFError:\n            logger.error(\n                \"Role %s rank %d is dead.\",\n                role_type.value,\n                rank_idx,\n            )\n            raise\n        if data.get(\"status\") != \"ready\":\n            raise RuntimeError(\n                f\"Role {role_type.value} rank {rank_idx} failed to initialize.\"\n            )\n        reader.close()\n\n    logger.info(\n        \"Role %s ready (%d GPU(s), work=%s)\",\n        role_type.value.upper(),\n        num_gpus,\n        work_endpoint,\n    )\n\n    # Block until interrupted\n    try:\n        for p in processes:\n            p.join()\n    except KeyboardInterrupt:\n        logger.info(\"Role %s shutting down.\", role_type.value)\n    finally:","sourceCodeStart":756,"sourceCodeEnd":792,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/launch_server.py#L756-L792","documentation":"Each disagg role process spawns GPU worker subprocesses and reads a readiness status over a pipe/socket. If a worker reports a status other than \"ready\" (or the earlier code path logs 'is dead' and re-raises), the launcher raises 'Role {role} rank {rank} failed to initialize', meaning a per-rank worker failed during model initialization (CUDA init, weight loading, NCCL setup).","triggerScenarios":"launch_disagg_role spawning N GPU workers; one rank crashes or reports status != 'ready' — e.g. CUDA OOM on one GPU, NCCL mismatch with tensor-parallel size, or an exception inside the worker before it sends the ready signal.","commonSituations":"Heterogeneous GPUs where one device has less free memory; tp/size larger than visible GPUs; NCCL/IPC issues in containers; one rank importing a missing dependency. The rank index in the message tells you which worker to inspect.","solutions":["Check per-rank worker logs/stack traces for the underlying exception on the reported rank.","Run `nvidia-smi` to confirm all ranks' GPUs are visible and have enough free memory for the model + TP shards.","Verify tensor-parallel size divides the available GPUs and NCCL env (e.g. master port) is set consistently across ranks.","Re-run with a smaller TP size or smaller model to isolate memory vs setup failures."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"import torch\ntotal = torch.cuda.device_count()\nassert server_args.tp_size <= total, f\"tp={server_args.tp_size} > visible GPUs={total}\"\nfree = [torch.cuda.mem_get_info(i)[0] for i in range(total)]\nassert min(free) > MIN_BYTES_NEEDED, \"not enough free VRAM on at least one rank\"","typeGuard":null,"tryCatchPattern":"try:\n    launch_disagg_role(server_args)\nexcept RuntimeError as e:\n    if \"failed to initialize\" in str(e):\n        # per-rank logs contain the root cause; retry after fixing memory/NCCL\n        raise","preventionTips":["Pre-flight check GPU count/free VRAM per rank before launch.","Pin NCCL socket/master settings consistently in multi-process scripts.","Capture each worker's stderr separately so rank failures are attributable."],"tags":["disaggregated-serving","multi-gpu","worker-init","nccl","tensor-parallel"],"backgroundTag":"worker-process-initialization-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}