{"record":{"id":"1e38156bba96c41e","repo":"languagetool-org/languagetool","slug":"more-than-2000-matches-for-term-not-supported","errorCode":null,"errorMessage":"More than 2000 matches for '${term}' not supported for performance reasons: ${docs.totalHits} matches in ${luceneSearcher.directory}","messagePattern":"More than 2000 matches for '(.+?)' not supported for performance reasons: (.+?) matches in (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java","lineNumber":198,"sourceCode":"    if (luceneSearcher == null) {\n      try {\n        LuceneSearcher newSearcher = new LuceneSearcher(indexDir);\n        dirToSearcherMap.put(indexDir, newSearcher);\n        return newSearcher;\n      } catch (IOException e) {\n        throw new RuntimeException(e);\n      }\n    } else {\n      return luceneSearcher;\n    }\n  }\n\n  private long getCount(Term term, LuceneSearcher luceneSearcher) {\n    long result = 0;\n    try {\n      TopDocs docs = luceneSearcher.searcher.search(new TermQuery(term), 2000);\n      if (docs.totalHits > 2000) {\n        throw new RuntimeException(\"More than 2000 matches for '\" + term + \"' not supported for performance reasons: \" +\n                                   docs.totalHits + \" matches in \" + luceneSearcher.directory);\n      }\n      for (ScoreDoc scoreDoc : docs.scoreDocs) {\n        String countStr = luceneSearcher.reader.document(scoreDoc.doc).get(\"count\");\n        result += Long.parseLong(countStr);\n      }\n      //System.out.println(term + \" -> \" + result);\n    } catch (IOException e) {\n      throw new RuntimeException(e);\n    }\n    return result;\n  }\n\n  @Override\n  public void close() {\n    for (LuceneSearcher searcher : luceneSearcherMap.values()) {\n      try {\n        searcher.reader.close();","sourceCodeStart":180,"sourceCodeEnd":216,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java#L180-L216","documentation":"getCount() searches for a Term in the ngram index but caps results at 2000 documents for performance. If the term matches more than 2000 documents the method aborts instead of accumulating counts. The index format is expected to store counts in few documents per term; exceeding the cap indicates an unexpectedly built or wrong index.","triggerScenarios":"Querying a term (via LanguageModel.getFrequency/getCount on a LuceneSingleIndexLanguageModel) where the TermQuery hits >2000 documents in the index, e.g. an index built without the count-aggregation layout LanguageTool expects, or querying against a foreign/general-purpose Lucene index.","commonSituations":"Pointing the model at an index built with different index settings, self-built indexes lacking the FrequencyIndexCreator doc structure, or indexes that contain many duplicate documents per term.","solutions":["Use the official LanguageTool ngram index for the language and Lucene version instead of a self-built or third-party index.","If self-building, run FrequencyIndexCreator so counts are aggregated into few documents per term.","Remove duplicates by rebuilding the index so each term maps to a small number of documents.","If you truly need huge counts, patch/raise the limit and use a higher search cap with awareness of memory cost."],"exampleFix":"// before: generic index built manually\nLanguageModel lm = new LuceneSingleIndexLanguageModel(Paths.get(\"/data/my-own-lucene-index\"));\n// after: proper LT index built with FrequencyIndexCreator\nLanguageModel lm = new LuceneSingleIndexLanguageModel(Paths.get(\"/data/ngrams/en\"));","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  long c = lm.getFrequency(tokens);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"More than 2000 matches\")) {\n    log.warn(\"Index incompatible with LT count layout\", e);\n  }\n}","preventionTips":["Use indexes built with FrequencyIndexCreator","Avoid pointing the model at foreign Lucene indexes","Rebuild indexes that contain duplicate term documents"],"tags":["lucene","ngram-index","performance"],"backgroundTag":"unsupported-operation","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}