{"record":{"id":"b02c733c3d104ce2","repo":"tensorflow/models","slug":"tokenize-with-offsets-is-not-supported-yet-when","errorCode":null,"errorMessage":"`tokenize_with_offsets` is not supported yet when `strip_diacritics` is set to True (b/181866850).","messagePattern":"`tokenize_with_offsets` is not supported yet when `strip_diacritics` is set to True \\(b/181866850\\)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"official/nlp/modeling/layers/text_layers.py","lineNumber":348,"sourceCode":"  def call(self, inputs: tf.Tensor):\n    \"\"\"Calls `text.SentencepieceTokenizer` on inputs.\n\n    Args:\n      inputs: A string Tensor of shape `(batch_size,)`.\n\n    Returns:\n      One or three of RaggedTensors if tokenize_with_offsets is False or True,\n      respectively. These are\n      tokens: A RaggedTensor of shape `[batch_size, (pieces)]` and type `int32`.\n        `tokens[i,j]` contains the j-th piece in the i-th input.\n      start_offsets, limit_offsets: If `tokenize_with_offsets` is True,\n        RaggedTensors of type `int64` with the same indices as tokens.\n        Element `[i,j]` contains the byte offset at the start, or past the\n        end, resp., for the j-th piece in the i-th input.\n    \"\"\"\n    if self._strip_diacritics:\n      if self.tokenize_with_offsets:\n        raise ValueError(\"`tokenize_with_offsets` is not supported yet when \"\n                         \"`strip_diacritics` is set to True (b/181866850).\")\n      inputs = text.normalize_utf8(inputs, \"NFD\")  # pyrefly: ignore[missing-attribute]\n      inputs = tf.strings.regex_replace(inputs, r\"\\p{Mn}\", \"\")\n\n    if self._lower_case:\n      inputs = text.case_fold_utf8(inputs)  # pyrefly: ignore[missing-attribute]\n\n    # Prepare to reshape the result to work around broken shape inference.\n    batch_size = tf.shape(inputs)[0]\n    def _reshape(rt):\n      values = rt.values\n      row_splits = rt.row_splits\n      row_splits = tf.reshape(row_splits, [batch_size + 1])\n      return tf.RaggedTensor.from_row_splits(values, row_splits)\n\n    # Call the tokenizer.\n    if self.tokenize_with_offsets:\n      tokens, start_offsets, limit_offsets = (","sourceCodeStart":330,"sourceCodeEnd":366,"githubUrl":"https://github.com/tensorflow/models/blob/e006f5f0d534913e49c1f1dae87364039fa607e2/official/nlp/modeling/layers/text_layers.py#L330-L366","documentation":"Error \"`tokenize_with_offsets` is not supported yet when `strip_diacritics` is set to True (b/181866850).\" thrown in tensorflow/models.","triggerScenarios":"Thrown at official/nlp/modeling/layers/text_layers.py:348 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":[],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"e006f5f0d534913e49c1f1dae87364039fa607e2","analyzedAt":"2026-08-24T14:09:15.576Z","schemaVersion":2},"datasetVersion":"2026-08-24T17:17:21.512Z"}