{"record":{"id":"10ac0b178502411b","repo":"rohitg00/ai-engineering-from-scratch","slug":"reduce-scatter-needs-numel-divisible-by-world-size","errorCode":null,"errorMessage":"reduce_scatter needs numel divisible by world_size, got {n} / {w}","messagePattern":"reduce_scatter needs numel divisible by world_size, got (.+?) / (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"phases/19-capstone-projects/76-collective-ops-from-scratch/code/main.py","lineNumber":184,"sourceCode":"        mesh.send(next_rank, shards[send_idx])\n        shards[recv_idx] = mesh.recv(prev_rank)\n    return torch.cat(shards)\n\n\ndef reduce_scatter(mesh: Mesh, tensor: torch.Tensor) -> torch.Tensor:\n    \"\"\"Reduce-scatter as the first half of ring allreduce.\n\n    Input is a tensor of length world_size * T. Output is the rank's chunk of\n    length T holding the sum across all ranks for that index range. The\n    underlying ring algorithm parks the full sum at index (r + 1) % W; we\n    return that chunk and label it as rank r's output to match\n    torch.distributed's contract that rank r owns chunks[r].\n    \"\"\"\n    w = mesh.world_size\n    r = mesh.rank\n    n = tensor.numel()\n    if n % w != 0:\n        raise ValueError(f\"reduce_scatter needs numel divisible by world_size, got {n} / {w}\")\n    if w == 1:\n        return tensor.clone()\n    rotated = list(tensor.chunk(w))\n    rotated = [rotated[(i - 1) % w].clone() for i in range(w)]\n    chunks = rotated\n    next_rank = (r + 1) % w\n    prev_rank = (r - 1) % w\n    for step in range(w - 1):\n        send_idx = (r - step) % w\n        recv_idx = (r - step - 1) % w\n        mesh.send(next_rank, chunks[send_idx])\n        incoming = mesh.recv(prev_rank)\n        chunks[recv_idx] = chunks[recv_idx] + incoming\n    return chunks[(r + 1) % w]\n\n\ndef _gloo_worker(rank: int, world_size: int, op: str, tensor_bytes: bytes,\n                 shape, dtype_str: str, init_file: str,","sourceCodeStart":166,"sourceCodeEnd":202,"githubUrl":"https://github.com/rohitg00/ai-engineering-from-scratch/blob/39ea8a1c6d0b61f071226eff7ede4d4105fed820/phases/19-capstone-projects/76-collective-ops-from-scratch/code/main.py#L166-L202","documentation":"Error \"reduce_scatter needs numel divisible by world_size, got {n} / {w}\" thrown in rohitg00/ai-engineering-from-scratch.","triggerScenarios":"Thrown at phases/19-capstone-projects/76-collective-ops-from-scratch/code/main.py:184 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":[],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"39ea8a1c6d0b61f071226eff7ede4d4105fed820","analyzedAt":"2026-08-26T03:13:46.626Z","schemaVersion":2},"datasetVersion":"2026-08-26T07:17:17.940Z"}