{"record":{"id":"42ec274426559d56","repo":"languagetool-org/languagetool","slug":"did-not-expect-more-than-1000-totaltokencount-me","errorCode":null,"errorMessage":"Did not expect more than 1000 'totalTokenCount' meta documents: ${docs.totalHits} in ${luceneSearcher.directory}","messagePattern":"Did not expect more than 1000 'totalTokenCount' meta documents: (.+?) in (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java","lineNumber":152,"sourceCode":"  public long getCount(String token1) {\n    Objects.requireNonNull(token1);\n    //TODO: move this into the document? It's not there currently...\n    //if (token1.equals(LanguageModel.GOOGLE_SENTENCE_START)) {\n    //  return 42_107_029_039L;  // see StartTokenCounter, run with 2grams (3grams: 124_541_229_392)\n    //}\n    return getCount(Arrays.asList(token1));\n  }\n\n  @Override\n  public long getTotalTokenCount() {\n    LuceneSearcher luceneSearcher = getLuceneSearcher(1);\n    try {\n      RegexpQuery query = new RegexpQuery(new Term(\"totalTokenCount\", \".*\"));\n      TopDocs docs = luceneSearcher.searcher.search(query, 1000);  // Integer.MAX_VALUE might cause OOE on wrong index\n      if (docs.totalHits == 0) {\n        throw new RuntimeException(\"Expected 'totalTokenCount' meta documents not found in 1grams index: \" + luceneSearcher.directory);\n      } else if (docs.totalHits > 1000) {\n        throw new RuntimeException(\"Did not expect more than 1000 'totalTokenCount' meta documents: \" + docs.totalHits + \" in \" + luceneSearcher.directory);\n      } else {\n        long result = 0;\n        for (ScoreDoc scoreDoc : docs.scoreDocs) {\n          long tmp = Long.parseLong(luceneSearcher.reader.document(scoreDoc.doc).get(\"totalTokenCount\"));\n          if (tmp > result) {\n            // due to the way FrequencyIndexCreator adds these totalTokenCount fields, we must not sum them,\n            // but take the largest one:\n            result = tmp;\n          }\n        }\n        return result;\n      }\n    } catch (IOException e) {\n      throw new RuntimeException(e);\n    }\n  }\n\n  protected LuceneSearcher getLuceneSearcher(int ngramSize) {","sourceCodeStart":134,"sourceCodeEnd":170,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java#L134-L170","documentation":"Thrown by getTotalTokenCount() when a RegexpQuery over the 'totalTokenCount' meta field in the Lucene ngram index returns more than 1000 documents. LanguageTool expects only a handful of meta documents holding the corpus total token count; a large hit count means the index is corrupt, wrongly built, or not a LanguageTool ngram index at all. The limit also protects memory, since an arbitrarily large search could OOM.","triggerScenarios":"Calling getTotalTokenCount() on a LuceneSingleIndexLanguageModel whose underlying index directory contains >1000 documents with a 'totalTokenCount' field, e.g. a wrong or corrupted 1grams index directory, or an index built by a different/incompatible FrequencyIndexCreator.","commonSituations":"Pointing ngram.languageModel at the wrong directory (top-level dir instead of the per-ngrams dirs), using an index built by a different Lucene LanguageTool version, or a partially copied/downloaded index that got mixed with other documents.","solutions":["Point the language model at the correct top-level directory that contains the 1grams/2grams/3grams subdirectories shipped for your LanguageTool version.","Re-download the ngram index archive for your LanguageTool version and verify its checksum.","Open the index with Luke or a small Lucene tool and confirm how many totalTokenCount documents exist; rebuild the index with LanguageTool's FrequencyIndexCreator if you built it yourself.","Check LanguageTool version compatibility: index format changed between versions; use the matching index release."],"exampleFix":"// before\nLanguageModel lm = new LuceneSingleIndexLanguageModel(Paths.get(\"/data/ngrams/wrong-dir\"));\n// after\nLanguageModel lm = new LuceneSingleIndexLanguageModel(Paths.get(\"/data/ngrams/en\")); // contains 1grams..3grams","handlingStrategy":"validation","validationCode":"File dir = Paths.get(\"/data/ngrams/en\").toFile();\nif (dir == null || !new File(dir, \"1grams\").isDirectory())\n  throw new IllegalStateException(\"Not a LanguageTool ngram index: \" + dir);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use only official LanguageTool ngram index downloads matching your LT version","Verify archive checksums after download","Never point the model at a general-purpose Lucene index"],"tags":["lucene","ngram-index","corrupt-index"],"backgroundTag":"invalid-config-value","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}