languagetool-org/languagetool · error · RuntimeException

Token not found: '%s' in tokens %s

Error message

Token not found: '%s' in tokens %s

What it means

LanguageModelUtils.getContext builds the token window (left/right context) around a given token for n-gram probability lookups. It locates the target token with List.indexOf; if the token is absent from the token list it throws this RuntimeException, because a context window cannot be computed without a position.

Source

Thrown at languagetool-core/src/main/java/org/languagetool/rules/ngrams/LanguageModelUtils.java:68

    return getContext(token, tokens, Collections.singletonList(new GoogleToken(newToken, 0, newToken.length())), toLeft, toRight);
  }


  static List<String> getContext(GoogleToken token, List<GoogleToken> tokens, List<GoogleToken> newTokens, int toLeft, int toRight) {
    List<GoogleToken> result = getContext(token, tokens, newTokens, toLeft, toRight,
      GoogleToken::isWhitespace, new GoogleToken(".", 0, 0));
    return result.stream().map(t -> t.token).collect(Collectors.toList());
  }

  public static <T> List<T> getContext(T token, List<T> tokens, List<T> newTokens, int toLeft, int toRight, Predicate<T> isWhitespace, T endToken) {
    // TODO: debug token not found sometimes
    //int pos = -1;
    //for (int i = 0; i < tokens.size(); i++) {
    //  if (tokens.get(i).token.s)
    //}
    int pos = tokens.indexOf(token);
    if (pos == -1) {
      throw new RuntimeException(String.format("Token not found: '%s' in tokens %s", token, tokens));
    }
    List<T> result = new ArrayList<T>();
    for (int i = 1, added = 0; added < toLeft; i++) {
      if (pos - i < 0) {
        // So if we're at the beginning of the sentence, just use the first tokens:
        result.clear();
        result.addAll(newTokens);
        for (int j = pos - 1; j >= 0; j--) {
          result.add(0, tokens.get(j));
        }
        return result;
      } else {
        if (!isWhitespace.test(tokens.get(pos - i))) {
          result.add(0, tokens.get(pos - i));
          added++;
        }
      }
    }

View on GitHub (pinned to 2e990059ce)

Solutions

1. Verify the token passed to getContext exists verbatim in the tokens list (same case and surface form)
2. Print/log the tokens list in the exception message and compare against the queried token
3. If the token may be absent, check tokens.contains(token) before calling getContext and skip the lookup
4. If token position is already known, refactor to pass pos directly instead of relying on indexOf

Example fix

// before
List<String> context = LanguageModelUtils.getContext(token, tokens, 2, 2, language);
// after
if (tokens.contains(token)) {
  List<String> context = LanguageModelUtils.getContext(token, tokens, 2, 2, language);
} else {
  // skip ngram check for this token
}

if (!tokens.contains(token)) {
  throw new IllegalArgumentException("Token '" + token + "' missing from token list");
}

boolean tokenPresent(String token, List<String> tokens) {
  return token != null && tokens != null && tokens.contains(token);
}

try {
  ctx = LanguageModelUtils.getContext(token, tokens, 2, 2, language);
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("Token not found")) {
    log.warn("Skipping ngram check; token not in list: {}", token);
    ctx = null;
  } else { throw e; }
}

When it happens

Trigger: Calling LanguageModelUtils.getContext (via result) with a token string that is not literally contained in the token list — e.g. the token was normalized/stemmed differently, case differs, or the token was removed from the list before the call.

Common situations: Rule code passing an AnalyzedToken text that differs from the raw token list entries, off-by-one preprocessing, or sentences where a token was split/merged before the language-model lookup.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Only 3grams and 4grams are supported
• Could not load language model capable rules.
• List of language models is empty
• Probability must be >= 0:
• No hits for

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/88fdd5800ab2f6ed. Report an issue: GitHub.