{"record":{"id":"c60488b8bac9353b","repo":"languagetool-org/languagetool","slug":"requested-tokens-size-gram-but-index-has-only","errorCode":null,"errorMessage":"Requested ${tokens.size()}gram but index has only up to ${maxNgram}gram: ${tokens}","messagePattern":"Requested (.+?)gram but index has only up to (.+?)gram: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java","lineNumber":126,"sourceCode":"  protected void doValidateDirectory(File topIndexDir) {\n    validateDirectory(topIndexDir);\n  }\n\n  private void addIndex(File topIndexDir, int ngramSize) {\n    File indexDir = new File(topIndexDir, ngramSize + \"grams\");\n    if (indexDir.exists() && indexDir.isDirectory()) {\n      if (luceneSearcherMap.containsKey(ngramSize)) {\n        throw new RuntimeException(\"Searcher for ngram size \" + ngramSize + \" already exists\");\n      }\n      luceneSearcherMap.put(ngramSize, getCachedLuceneSearcher(indexDir));\n      indexes.add(indexDir);\n    }\n  }\n\n  @Override\n  public long getCount(List<String> tokens) {\n    if (tokens.size() > maxNgram) {\n      throw new RuntimeException(\"Requested \" + tokens.size() + \"gram but index has only up to \" + maxNgram + \"gram: \" + tokens);\n    }\n    Objects.requireNonNull(tokens);\n    Term term = new Term(\"ngram\", String.join(\" \", tokens));\n    return getCount(term, getLuceneSearcher(tokens.size()));\n  }\n\n  @Override\n  public long getCount(String token1) {\n    Objects.requireNonNull(token1);\n    //TODO: move this into the document? It's not there currently...\n    //if (token1.equals(LanguageModel.GOOGLE_SENTENCE_START)) {\n    //  return 42_107_029_039L;  // see StartTokenCounter, run with 2grams (3grams: 124_541_229_392)\n    //}\n    return getCount(Arrays.asList(token1));\n  }\n\n  @Override\n  public long getTotalTokenCount() {","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java#L108-L144","documentation":"getCount(List<String> tokens) can only answer for ngrams up to the index's maxNgram size. Requesting a longer token list throws RuntimeException stating the requested ngram size and the index maximum.","triggerScenarios":"Calling getCount() with a List<String> longer than maxNgram, e.g. a 5-word list against a 3gram index built from the standard 1grams-3grams data.","commonSituations":"Code that slices variable-length windows from text without capping the window at the model's ngram size; using a language model trained with fewer ngram levels than the caller assumes.","solutions":["Truncate the token list to maxNgram before calling getCount()","Query in chunks: sum/log-combine counts of overlapping maxNgram-sized sub-ngrams","Provide an index with a higher maxNgram if longer ngrams are genuinely needed"],"exampleFix":"// before\nlong c = lm.getCount(fiveTokens); // RuntimeException on 3gram index\n// after\nlong c = lm.getCount(fiveTokens.subList(0, lm.getMaxNgram()));","handlingStrategy":"validation","validationCode":"if (tokens.size() > lm.getMaxNgram()) { tokens = tokens.subList(0, lm.getMaxNgram()); }","typeGuard":null,"tryCatchPattern":"try { long c = lm.getCount(tokens); } catch (RuntimeException e) { if (e.getMessage().contains(\"gram but index has only\")) { c = lm.getCount(tokens.subList(0, lm.getMaxNgram())); } else throw e; }","preventionTips":["Cap ngram window sizes in your code at the model's maxNgram","Check getMaxNgram() once at startup and store it for all callers","When switching indexes, recompute window sizes instead of hardcoding 3 or 5"],"tags":["java","lucene","ngrams","argument-out-of-range"],"backgroundTag":"value-out-of-range","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}