{"record":{"id":"1333408ad52c3ff0","repo":"babysor/MockingBird","slug":"batch-size-must-be-evenly-divisible-by-n-gpus","errorCode":null,"errorMessage":"`batch_size` must be evenly divisible by n_gpus!","messagePattern":"`batch_size` must be evenly divisible by n_gpus!","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"models/synthesizer/train.py","lineNumber":64,"sourceCode":"    \n    print(\"Checkpoint path: {}\".format(weights_fpath))\n    print(\"Loading training data from: {}\".format(metadata_fpath))\n    print(\"Using model: Tacotron\")\n    \n    # Book keeping\n    step = 0\n    time_window = ValueWindow(100)\n    loss_window = ValueWindow(100)\n    \n    \n    # From WaveRNN/train_tacotron.py\n    if torch.cuda.is_available():\n        device = torch.device(\"cuda\")\n\n        for session in hparams.tts_schedule:\n            _, _, _, batch_size = session\n            if batch_size % torch.cuda.device_count() != 0:\n                raise ValueError(\"`batch_size` must be evenly divisible by n_gpus!\")\n    else:\n        device = torch.device(\"cpu\")\n    print(\"Using device:\", device)\n\n    # Instantiate Tacotron Model\n    print(\"\\nInitialising Tacotron Model...\\n\")\n    num_chars = len(symbols)\n    if weights_fpath.exists():\n        # for compatibility purpose, change symbols accordingly:\n        loaded_shape = torch.load(str(weights_fpath), map_location=device)[\"model_state\"][\"encoder.embedding.weight\"].shape\n        if num_chars != loaded_shape[0]:\n            print(\"WARNING: you are using compatible mode due to wrong sympols length, please modify varible _characters in `utils\\symbols.py`\")\n            num_chars != loaded_shape[0]\n                # Try to scan config file\n        model_config_fpaths = list(weights_fpath.parent.rglob(\"*.json\"))\n        if len(model_config_fpaths)>0 and model_config_fpaths[0].exists():\n            hparams.loadJson(model_config_fpaths[0])\n        else:  # save a config","sourceCodeStart":46,"sourceCodeEnd":82,"githubUrl":"https://github.com/babysor/MockingBird/blob/28dc5e14f12d7c754612af2fde8e78a4b03f8616/models/synthesizer/train.py#L46-L82","documentation":"The Tacotron training loop checks every entry of hparams.tts_schedule and requires each session's batch_size to be divisible by the number of CUDA devices, since DataParallel splits the batch across GPUs.","triggerScenarios":"Calling train() on a multi-GPU machine where any (learning_rate, ...) tuple in tts_schedule has a batch_size not divisible by torch.cuda.device_count().","commonSituations":"Using the default DeepVoice3/LJSpeech schedule (batch sizes like 1-16 ramps) on an 8-GPU box; changing GPU count without updating the schedule.","solutions":["Edit hparams.tts_schedule so every session's batch_size is a multiple of n_gpus","Or pin to one GPU with CUDA_VISIBLE_DEVICES=0","Multiply all scheduled batch sizes by device_count and scale LR accordingly if needed"],"exampleFix":"# before\ntts_schedule: [[...], [0.0001, 1e-6, 10000, 3]]  # n_gpus=2 → 3 % 2 != 0\n\n# after\ntts_schedule: [[...], [0.0001, 1e-6, 10000, 4]]","handlingStrategy":"validation","validationCode":"n = torch.cuda.device_count()\nfor session in hparams.tts_schedule:\n    _, _, _, bs = session\n    assert n == 0 or bs % n == 0, f'batch_size {bs} not divisible by {n} GPUs'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate the whole tts_schedule up front","Pin single GPU for small-batch training schedules"],"tags":["multi-gpu","batch-size","dataparallel","training-schedule"],"backgroundTag":"batch-size-not-divisible-by-gpu-count","analyzedSha":"28dc5e14f12d7c754612af2fde8e78a4b03f8616","analyzedAt":"2026-08-27T02:26:53.589Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}