tensorflow/models · error · ValueError

`tokenize_with_offsets` is not supported yet when `strip_dia

Error message

`tokenize_with_offsets` is not supported yet when `strip_diacritics` is set to True (b/181866850).

What it means

Error "`tokenize_with_offsets` is not supported yet when `strip_diacritics` is set to True (b/181866850)." thrown in tensorflow/models.

Source

Thrown at official/nlp/modeling/layers/text_layers.py:348

  def call(self, inputs: tf.Tensor):
    """Calls `text.SentencepieceTokenizer` on inputs.

    Args:
      inputs: A string Tensor of shape `(batch_size,)`.

    Returns:
      One or three of RaggedTensors if tokenize_with_offsets is False or True,
      respectively. These are
      tokens: A RaggedTensor of shape `[batch_size, (pieces)]` and type `int32`.
        `tokens[i,j]` contains the j-th piece in the i-th input.
      start_offsets, limit_offsets: If `tokenize_with_offsets` is True,
        RaggedTensors of type `int64` with the same indices as tokens.
        Element `[i,j]` contains the byte offset at the start, or past the
        end, resp., for the j-th piece in the i-th input.
    """
    if self._strip_diacritics:
      if self.tokenize_with_offsets:
        raise ValueError("`tokenize_with_offsets` is not supported yet when "
                         "`strip_diacritics` is set to True (b/181866850).")
      inputs = text.normalize_utf8(inputs, "NFD")  # pyrefly: ignore[missing-attribute]
      inputs = tf.strings.regex_replace(inputs, r"\p{Mn}", "")

    if self._lower_case:
      inputs = text.case_fold_utf8(inputs)  # pyrefly: ignore[missing-attribute]

    # Prepare to reshape the result to work around broken shape inference.
    batch_size = tf.shape(inputs)[0]
    def _reshape(rt):
      values = rt.values
      row_splits = rt.row_splits
      row_splits = tf.reshape(row_splits, [batch_size + 1])
      return tf.RaggedTensor.from_row_splits(values, row_splits)

    # Call the tokenizer.
    if self.tokenize_with_offsets:
      tokens, start_offsets, limit_offsets = (

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/modeling/layers/text_layers.py:348 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/b02c733c3d104ce2. Report an issue: GitHub.