tensorflow/models · error · ValueError

EOS token not in tokenizer vocab.Please make sure the tokeni

Error message

EOS token not in tokenizer vocab.Please make sure the tokenizer generates a single token for an empty string.

What it means

Error "EOS token not in tokenizer vocab.Please make sure the tokenizer generates a single token for an empty string." thrown in tensorflow/models.

Source

Thrown at official/nlp/serving/serving_modules.py:431

class Translation(export_base.ExportModule):
  """The export module for the translation task."""

  @dataclasses.dataclass
  class Params(base_config.Config):
    sentencepiece_model_path: str = ""
    # Needs to be specified if padded_decode is True/on TPUs.
    batch_size: Optional[int] = None

  def __init__(self, params, model: tf_keras.Model, inference_step=None):
    super().__init__(params, model, inference_step)
    self._sp_tokenizer = tf_text.SentencepieceTokenizer(
        model=tf.io.gfile.GFile(params.sentencepiece_model_path, "rb").read(),
        add_eos=True)
    try:
      empty_str_tokenized = self._sp_tokenizer.tokenize("").numpy()
    except tf.errors.InternalError:
      raise ValueError(
          "EOS token not in tokenizer vocab."
          "Please make sure the tokenizer generates a single token for an "
          "empty string.")
    self._eos_id = empty_str_tokenized.item()
    self._batch_size = params.batch_size

  @tf.function
  def serve(self, inputs) -> Dict[str, tf.Tensor]:
    return self.inference_step(inputs)

  @tf.function
  def serve_text(self, text: tf.Tensor) -> Dict[str, tf.Tensor]:
    tokenized = self._sp_tokenizer.tokenize(text).to_tensor(0)
    return self._sp_tokenizer.detokenize(
        self.serve({"inputs": tokenized})["outputs"])

  def get_inference_signatures(self, function_keys: Dict[Text, Text]):
    signatures = {}

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/serving/serving_modules.py:431 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/02f23e5142e2f606. Report an issue: GitHub.