languagetool-org/languagetool · error · RuntimeException

Requested ${tokens.size()}gram but index has only up to ${ma

Error message

Requested ${tokens.size()}gram but index has only up to ${maxNgram}gram: ${tokens}

What it means

getCount(List<String> tokens) can only answer for ngrams up to the index's maxNgram size. Requesting a longer token list throws RuntimeException stating the requested ngram size and the index maximum.

Source

Thrown at languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java:126

  protected void doValidateDirectory(File topIndexDir) {
    validateDirectory(topIndexDir);
  }

  private void addIndex(File topIndexDir, int ngramSize) {
    File indexDir = new File(topIndexDir, ngramSize + "grams");
    if (indexDir.exists() && indexDir.isDirectory()) {
      if (luceneSearcherMap.containsKey(ngramSize)) {
        throw new RuntimeException("Searcher for ngram size " + ngramSize + " already exists");
      }
      luceneSearcherMap.put(ngramSize, getCachedLuceneSearcher(indexDir));
      indexes.add(indexDir);
    }
  }

  @Override
  public long getCount(List<String> tokens) {
    if (tokens.size() > maxNgram) {
      throw new RuntimeException("Requested " + tokens.size() + "gram but index has only up to " + maxNgram + "gram: " + tokens);
    }
    Objects.requireNonNull(tokens);
    Term term = new Term("ngram", String.join(" ", tokens));
    return getCount(term, getLuceneSearcher(tokens.size()));
  }

  @Override
  public long getCount(String token1) {
    Objects.requireNonNull(token1);
    //TODO: move this into the document? It's not there currently...
    //if (token1.equals(LanguageModel.GOOGLE_SENTENCE_START)) {
    //  return 42_107_029_039L;  // see StartTokenCounter, run with 2grams (3grams: 124_541_229_392)
    //}
    return getCount(Arrays.asList(token1));
  }

  @Override
  public long getTotalTokenCount() {

View on GitHub (pinned to 2e990059ce)

Solutions

Truncate the token list to maxNgram before calling getCount()
Query in chunks: sum/log-combine counts of overlapping maxNgram-sized sub-ngrams
Provide an index with a higher maxNgram if longer ngrams are genuinely needed

Example fix

// before
long c = lm.getCount(fiveTokens); // RuntimeException on 3gram index
// after
long c = lm.getCount(fiveTokens.subList(0, lm.getMaxNgram()));

Defensive patterns

Strategy: validation

Validate before calling

if (tokens.size() > lm.getMaxNgram()) { tokens = tokens.subList(0, lm.getMaxNgram()); }

Try / catch

try { long c = lm.getCount(tokens); } catch (RuntimeException e) { if (e.getMessage().contains("gram but index has only")) { c = lm.getCount(tokens.subList(0, lm.getMaxNgram())); } else throw e; }

Prevention

Cap ngram window sizes in your code at the model's maxNgram
Check getMaxNgram() once at startup and store it for all callers
When switching indexes, recompute window sizes instead of hardcoding 3 or 5

When it happens

Trigger: Calling getCount() with a List<String> longer than maxNgram, e.g. a 5-word list against a 3gram index built from the standard 1grams-3grams data.

Common situations: Code that slices variable-length windows from text without capping the window at the model's ngram size; using a language model trained with fewer ngram levels than the caller assumes.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/c60488b8bac9353b. Report an issue: GitHub.