languagetool-org/languagetool · error · RuntimeException

More than 2000 matches for '${term}' not supported for perfo

Error message

More than 2000 matches for '${term}' not supported for performance reasons: ${docs.totalHits} matches in ${luceneSearcher.directory}

What it means

getCount() searches for a Term in the ngram index but caps results at 2000 documents for performance. If the term matches more than 2000 documents the method aborts instead of accumulating counts. The index format is expected to store counts in few documents per term; exceeding the cap indicates an unexpectedly built or wrong index.

Source

Thrown at languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java:198

    if (luceneSearcher == null) {
      try {
        LuceneSearcher newSearcher = new LuceneSearcher(indexDir);
        dirToSearcherMap.put(indexDir, newSearcher);
        return newSearcher;
      } catch (IOException e) {
        throw new RuntimeException(e);
      }
    } else {
      return luceneSearcher;
    }
  }

  private long getCount(Term term, LuceneSearcher luceneSearcher) {
    long result = 0;
    try {
      TopDocs docs = luceneSearcher.searcher.search(new TermQuery(term), 2000);
      if (docs.totalHits > 2000) {
        throw new RuntimeException("More than 2000 matches for '" + term + "' not supported for performance reasons: " +
                                   docs.totalHits + " matches in " + luceneSearcher.directory);
      }
      for (ScoreDoc scoreDoc : docs.scoreDocs) {
        String countStr = luceneSearcher.reader.document(scoreDoc.doc).get("count");
        result += Long.parseLong(countStr);
      }
      //System.out.println(term + " -> " + result);
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
    return result;
  }

  @Override
  public void close() {
    for (LuceneSearcher searcher : luceneSearcherMap.values()) {
      try {
        searcher.reader.close();

View on GitHub (pinned to 2e990059ce)

Solutions

Use the official LanguageTool ngram index for the language and Lucene version instead of a self-built or third-party index.
If self-building, run FrequencyIndexCreator so counts are aggregated into few documents per term.
Remove duplicates by rebuilding the index so each term maps to a small number of documents.
If you truly need huge counts, patch/raise the limit and use a higher search cap with awareness of memory cost.

Example fix

// before: generic index built manually
LanguageModel lm = new LuceneSingleIndexLanguageModel(Paths.get("/data/my-own-lucene-index"));
// after: proper LT index built with FrequencyIndexCreator
LanguageModel lm = new LuceneSingleIndexLanguageModel(Paths.get("/data/ngrams/en"));

Defensive patterns

Strategy: try-catch

Try / catch

try {
  long c = lm.getFrequency(tokens);
} catch (RuntimeException e) {
  if (e.getMessage().contains("More than 2000 matches")) {
    log.warn("Index incompatible with LT count layout", e);
  }
}

Prevention

Use indexes built with FrequencyIndexCreator
Avoid pointing the model at foreign Lucene indexes
Rebuild indexes that contain duplicate term documents

When it happens

Trigger: Querying a term (via LanguageModel.getFrequency/getCount on a LuceneSingleIndexLanguageModel) where the TermQuery hits >2000 documents in the index, e.g. an index built without the count-aggregation layout LanguageTool expects, or querying against a foreign/general-purpose Lucene index.

Common situations: Pointing the model at an index built with different index settings, self-built indexes lacking the FrequencyIndexCreator doc structure, or indexes that contain many duplicate documents per term.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/1e38156bba96c41e. Report an issue: GitHub.