tensorflow/models · error · ValueError

`tokenize_with_offsets` is not supported when `strip_diacrit

Error message

`tokenize_with_offsets` is not supported when `strip_diacritics` is set to True.

What it means

Error "`tokenize_with_offsets` is not supported when `strip_diacritics` is set to True." thrown in tensorflow/models.

Source

Thrown at official/nlp/modeling/layers/text_layers.py:303

        diacritics requires additional text normalization and dropping bytes,
        which makes it impossible to keep track of the offsets now. Hence
        when `strip_diacritics` is set to True, we don't yet support
        `tokenize_with_offsets`. NOTE: New models are encouraged to put this
        into custom normalization rules for the Sentencepiece model itself to
        avoid this extra step and the limitation regarding offsets.
      **kwargs: standard arguments to `Layer()`.

    Raises:
      ImportError: if importing tensorflow_text failed.
    """
    _check_if_tf_text_installed()
    super().__init__(**kwargs)
    if bool(model_file_path) == bool(model_serialized_proto):
      raise ValueError("Exact one of `model_file_path` and "
                       "`model_serialized_proto` can be specified.")
    # TODO(b/181866850): Support tokenize_with_offsets for strip_diacritics=True
    if tokenize_with_offsets and strip_diacritics:
      raise ValueError("`tokenize_with_offsets` is not supported when "
                       "`strip_diacritics` is set to True.")
    if model_file_path:
      self._model_serialized_proto = tf.io.gfile.GFile(model_file_path,
                                                       "rb").read()
    else:
      self._model_serialized_proto = model_serialized_proto

    self._lower_case = lower_case
    self.tokenize_with_offsets = tokenize_with_offsets
    self._nbest_size = nbest_size
    self._alpha = alpha
    self._strip_diacritics = strip_diacritics
    self._tokenizer = self._create_tokenizer()
    self._special_tokens_dict = self._create_special_tokens_dict()

  def _create_tokenizer(self):
    return text.SentencepieceTokenizer(  # pyrefly: ignore[missing-attribute]
        model=self._model_serialized_proto,

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/modeling/layers/text_layers.py:303 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/9cd5a41939bf0bd7. Report an issue: GitHub.