tensorflow/models · error · ValueError

cannot use ngram masking without whole word masking

Error message

cannot use ngram masking without whole word masking

What it means

Error "cannot use ngram masking without whole word masking" thrown in tensorflow/models.

Source

Thrown at official/nlp/data/create_pretraining_data.py:597

      gram_start_pos = i
    else:
      gram_start_pos = None
  if gram_start_pos is not None:
    grams.append(_Gram(gram_start_pos, len(tokens)))
  return grams


def create_masked_lm_predictions(tokens, masked_lm_prob,
                                 max_predictions_per_seq, vocab_words, rng,
                                 do_whole_word_mask,
                                 max_ngram_size=None):
  """Creates the predictions for the masked LM objective."""
  if do_whole_word_mask:
    grams = _tokens_to_grams(tokens)
  else:
    # Here we consider each token to be a word to allow for sub-word masking.
    if max_ngram_size:
      raise ValueError("cannot use ngram masking without whole word masking")
    grams = [_Gram(i, i+1) for i in range(0, len(tokens))
             if tokens[i] not in ["[CLS]", "[SEP]"]]

  num_to_predict = min(max_predictions_per_seq,
                       max(1, int(round(len(tokens) * masked_lm_prob))))
  # Generate masks.  If `max_ngram_size` in [0, None] it means we're doing
  # whole word masking or token level masking.  Both of these can be treated
  # as the `max_ngram_size=1` case.
  masked_grams = _masking_ngrams(grams, max_ngram_size or 1,
                                 num_to_predict, rng)
  masked_lms = []
  output_tokens = list(tokens)
  for gram in masked_grams:
    # 80% of the time, replace all n-gram tokens with [MASK]
    if rng.random() < 0.8:
      replacement_action = lambda idx: "[MASK]"
    else:
      # 10% of the time, keep all the original n-gram tokens.

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/data/create_pretraining_data.py:597 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/86e5f88ce6de9856. Report an issue: GitHub.