tensorflow/models · error · ValueError

Unsupported tokenization: %s

Error message

Unsupported tokenization: %s

What it means

Error "Unsupported tokenization: %s" thrown in tensorflow/models.

Source

Thrown at official/nlp/data/sentence_prediction_dataloader.py:180

               preprocessing_hub_module_url: Optional[str] = None):
    if preprocessing_hub_module_url:
      self._preprocessing_hub_module = hub.load(preprocessing_hub_module_url)
      self._tokenizer = self._preprocessing_hub_module.tokenize
      self._pack_inputs = functools.partial(
          self._preprocessing_hub_module.bert_pack_inputs,
          seq_length=seq_length)
      return

    if tokenization == 'WordPiece':
      self._tokenizer = modeling.layers.BertTokenizer(
          vocab_file=vocab_file, lower_case=lower_case)
    elif tokenization == 'SentencePiece':
      self._tokenizer = modeling.layers.SentencepieceTokenizer(
          model_file_path=vocab_file,
          lower_case=lower_case,
          strip_diacritics=True)  # Strip diacritics to follow ALBERT model
    else:
      raise ValueError('Unsupported tokenization: %s' % tokenization)

    self._pack_inputs = modeling.layers.BertPackInputs(
        seq_length=seq_length,
        special_tokens_dict=self._tokenizer.get_special_tokens_dict())

  def __call__(self, segments):
    segments = [self._tokenizer(s) for s in segments]
    # BertTokenizer returns a RaggedTensor with shape [batch, word, subword],
    # and SentencepieceTokenizer returns a RaggedTensor with shape
    # [batch, sentencepiece],
    segments = [
        tf.cast(x.merge_dims(1, -1) if x.shape.rank > 2 else x, tf.int32)
        for x in segments
    ]
    return self._pack_inputs(segments)


@data_loader_factory.register_data_loader_cls(SentencePredictionTextDataConfig)

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/data/sentence_prediction_dataloader.py:180 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/ce741316161de226. Report an issue: GitHub.