{"record":{"id":"88fdd5800ab2f6ed","repo":"languagetool-org/languagetool","slug":"token-not-found-s-in-tokens-s","errorCode":null,"errorMessage":"Token not found: '%s' in tokens %s","messagePattern":"Token not found: '(.+?)' in tokens (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"languagetool-core/src/main/java/org/languagetool/rules/ngrams/LanguageModelUtils.java","lineNumber":68,"sourceCode":"    return getContext(token, tokens, Collections.singletonList(new GoogleToken(newToken, 0, newToken.length())), toLeft, toRight);\n  }\n\n\n  static List<String> getContext(GoogleToken token, List<GoogleToken> tokens, List<GoogleToken> newTokens, int toLeft, int toRight) {\n    List<GoogleToken> result = getContext(token, tokens, newTokens, toLeft, toRight,\n      GoogleToken::isWhitespace, new GoogleToken(\".\", 0, 0));\n    return result.stream().map(t -> t.token).collect(Collectors.toList());\n  }\n\n  public static <T> List<T> getContext(T token, List<T> tokens, List<T> newTokens, int toLeft, int toRight, Predicate<T> isWhitespace, T endToken) {\n    // TODO: debug token not found sometimes\n    //int pos = -1;\n    //for (int i = 0; i < tokens.size(); i++) {\n    //  if (tokens.get(i).token.s)\n    //}\n    int pos = tokens.indexOf(token);\n    if (pos == -1) {\n      throw new RuntimeException(String.format(\"Token not found: '%s' in tokens %s\", token, tokens));\n    }\n    List<T> result = new ArrayList<T>();\n    for (int i = 1, added = 0; added < toLeft; i++) {\n      if (pos - i < 0) {\n        // So if we're at the beginning of the sentence, just use the first tokens:\n        result.clear();\n        result.addAll(newTokens);\n        for (int j = pos - 1; j >= 0; j--) {\n          result.add(0, tokens.get(j));\n        }\n        return result;\n      } else {\n        if (!isWhitespace.test(tokens.get(pos - i))) {\n          result.add(0, tokens.get(pos - i));\n          added++;\n        }\n      }\n    }","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-core/src/main/java/org/languagetool/rules/ngrams/LanguageModelUtils.java#L50-L86","documentation":"LanguageModelUtils.getContext builds the token window (left/right context) around a given token for n-gram probability lookups. It locates the target token with List.indexOf; if the token is absent from the token list it throws this RuntimeException, because a context window cannot be computed without a position.","triggerScenarios":"Calling LanguageModelUtils.getContext (via result) with a token string that is not literally contained in the token list — e.g. the token was normalized/stemmed differently, case differs, or the token was removed from the list before the call.","commonSituations":"Rule code passing an AnalyzedToken text that differs from the raw token list entries, off-by-one preprocessing, or sentences where a token was split/merged before the language-model lookup.","solutions":["Verify the token passed to getContext exists verbatim in the tokens list (same case and surface form)","Print/log the tokens list in the exception message and compare against the queried token","If the token may be absent, check tokens.contains(token) before calling getContext and skip the lookup","If token position is already known, refactor to pass pos directly instead of relying on indexOf"],"exampleFix":"// before\nList<String> context = LanguageModelUtils.getContext(token, tokens, 2, 2, language);\n// after\nif (tokens.contains(token)) {\n  List<String> context = LanguageModelUtils.getContext(token, tokens, 2, 2, language);\n} else {\n  // skip ngram check for this token\n}","handlingStrategy":"validation","validationCode":"if (!tokens.contains(token)) {\n  throw new IllegalArgumentException(\"Token '\" + token + \"' missing from token list\");\n}","typeGuard":"boolean tokenPresent(String token, List<String> tokens) {\n  return token != null && tokens != null && tokens.contains(token);\n}","tryCatchPattern":"try {\n  ctx = LanguageModelUtils.getContext(token, tokens, 2, 2, language);\n} catch (RuntimeException e) {\n  if (e.getMessage().startsWith(\"Token not found\")) {\n    log.warn(\"Skipping ngram check; token not in list: {}\", token);\n    ctx = null;\n  } else { throw e; }\n}","preventionTips":["Pass raw surface tokens, never analyzed/stemmed forms","Keep token list and lookup token from the same AnalyzedTokenReadings sequence","Log the token list in the failure path to diagnose case/split mismatches"],"tags":["ngram","tokenization","language-model","index-lookup"],"backgroundTag":"resource-not-found","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}