{"record":{"id":"fc5c3a3de941ad7b","repo":"huggingface/transformers","slug":"the-sum-of-mask-replace-prob-and-random-replace-pr","errorCode":null,"errorMessage":"The sum of mask_replace_prob and random_replace_prob should not exceed 1","messagePattern":"The sum of mask_replace_prob and random_replace_prob should not exceed 1","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/data/data_collator.py","lineNumber":708,"sourceCode":"    seed: int | None = None\n\n    def __post_init__(self):\n        if self.mlm:\n            if self.tokenizer.mask_token is None:\n                raise ValueError(\n                    \"This tokenizer does not have a mask token which is necessary for masked language modeling. \"\n                    \"You should pass `mlm=False` to train on causal language modeling instead.\"\n                )\n            if self.mlm_probability is None or self.mlm_probability < 0 or self.mlm_probability > 1:\n                raise ValueError(\"mlm_probability should be between 0 and 1.\")\n            self.mlm_probability = float(self.mlm_probability)\n        elif self.whole_word_mask:\n            raise ValueError(\n                \"Whole word masking can only be used with mlm=True.\"\n                \"If you want to use whole word masking, please set mlm=True.\"\n            )\n        if self.mask_replace_prob + self.random_replace_prob > 1:\n            raise ValueError(\"The sum of mask_replace_prob and random_replace_prob should not exceed 1\")\n        if self.mask_replace_prob < 0 or self.mask_replace_prob > 1:\n            raise ValueError(\"mask_replace_prob should be between 0 and 1.\")\n        if self.random_replace_prob < 0 or self.random_replace_prob > 1:\n            raise ValueError(\"random_replace_prob should be between 0 and 1.\")\n\n        if self.whole_word_mask:\n            if not self.tokenizer.is_fast:\n                warnings.warn(\n                    \"Whole word masking depends on offset mapping which is only natively available with fast tokenizers.\",\n                    UserWarning,\n                )\n\n            if self.mask_replace_prob < 1:\n                warnings.warn(\n                    \"Random token replacement is not supported with whole word masking. \"\n                    \"Setting mask_replace_prob to 1.\",\n                )\n                self.mask_replace_prob = 1","sourceCodeStart":690,"sourceCodeEnd":726,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/data/data_collator.py#L690-L726","documentation":"Raised by DataCollatorForLanguageModeling.__post_init__ when mask_replace_prob + random_replace_prob > 1. The collator splits masked tokens into three disjoint groups (replaced by [MASK], replaced by a random token, left unchanged); if the two replacement probabilities sum above 1 there is no valid remainder, so the configuration is rejected.","triggerScenarios":"Constructing DataCollatorForLanguageModeling with e.g. mask_replace_prob=0.9 and random_replace_prob=0.2 (defaults are 0.8 and 0.1). Validated on every instantiation, before any batching happens.","commonSituations":"Tuning the 80/10/10 BERT masking ratios and forgetting the third 'unchanged' bucket; passing percentages (e.g. 80 and 30 instead of 0.8 and 0.3).","solutions":["Lower one or both probabilities so mask_replace_prob + random_replace_prob <= 1 (e.g. defaults 0.8 + 0.1).","If you want every masked token changed, use exactly mask_replace_prob=1.0, random_replace_prob=0.0 (or 0.0/1.0).","Double-check that you passed fractions in [0,1], not integer percentages."],"exampleFix":"# before\ncollator = DataCollatorForLanguageModeling(tokenizer=tok, mask_replace_prob=0.9, random_replace_prob=0.3)\n\n# after\ncollator = DataCollatorForLanguageModeling(tokenizer=tok, mask_replace_prob=0.8, random_replace_prob=0.2)","handlingStrategy":"validation","validationCode":"def check_mask_probs(mask_replace_prob, random_replace_prob):\n    assert 0 <= mask_replace_prob <= 1 and 0 <= random_replace_prob <= 1\n    assert mask_replace_prob + random_replace_prob <= 1, 'replacement probs must sum to <= 1'\n\ncheck_mask_probs(0.8, 0.2)  # call before constructing the collator","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Centralize masking-ratio hyperparameters in one config and assert the sum rule there.","Always express probabilities as fractions in [0,1], never percentages."],"tags":["data-collator","configuration","validation","masked-language-modeling"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}