languagetool-org/languagetool · critical · RuntimeException

Did not expect more than 1000 'totalTokenCount' meta documen

Error message

Did not expect more than 1000 'totalTokenCount' meta documents: ${docs.totalHits} in ${luceneSearcher.directory}

What it means

Thrown by getTotalTokenCount() when a RegexpQuery over the 'totalTokenCount' meta field in the Lucene ngram index returns more than 1000 documents. LanguageTool expects only a handful of meta documents holding the corpus total token count; a large hit count means the index is corrupt, wrongly built, or not a LanguageTool ngram index at all. The limit also protects memory, since an arbitrarily large search could OOM.

Source

Thrown at languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java:152

  public long getCount(String token1) {
    Objects.requireNonNull(token1);
    //TODO: move this into the document? It's not there currently...
    //if (token1.equals(LanguageModel.GOOGLE_SENTENCE_START)) {
    //  return 42_107_029_039L;  // see StartTokenCounter, run with 2grams (3grams: 124_541_229_392)
    //}
    return getCount(Arrays.asList(token1));
  }

  @Override
  public long getTotalTokenCount() {
    LuceneSearcher luceneSearcher = getLuceneSearcher(1);
    try {
      RegexpQuery query = new RegexpQuery(new Term("totalTokenCount", ".*"));
      TopDocs docs = luceneSearcher.searcher.search(query, 1000);  // Integer.MAX_VALUE might cause OOE on wrong index
      if (docs.totalHits == 0) {
        throw new RuntimeException("Expected 'totalTokenCount' meta documents not found in 1grams index: " + luceneSearcher.directory);
      } else if (docs.totalHits > 1000) {
        throw new RuntimeException("Did not expect more than 1000 'totalTokenCount' meta documents: " + docs.totalHits + " in " + luceneSearcher.directory);
      } else {
        long result = 0;
        for (ScoreDoc scoreDoc : docs.scoreDocs) {
          long tmp = Long.parseLong(luceneSearcher.reader.document(scoreDoc.doc).get("totalTokenCount"));
          if (tmp > result) {
            // due to the way FrequencyIndexCreator adds these totalTokenCount fields, we must not sum them,
            // but take the largest one:
            result = tmp;
          }
        }
        return result;
      }
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }

  protected LuceneSearcher getLuceneSearcher(int ngramSize) {

View on GitHub (pinned to 2e990059ce)

Solutions

Point the language model at the correct top-level directory that contains the 1grams/2grams/3grams subdirectories shipped for your LanguageTool version.
Re-download the ngram index archive for your LanguageTool version and verify its checksum.
Open the index with Luke or a small Lucene tool and confirm how many totalTokenCount documents exist; rebuild the index with LanguageTool's FrequencyIndexCreator if you built it yourself.
Check LanguageTool version compatibility: index format changed between versions; use the matching index release.

Example fix

// before
LanguageModel lm = new LuceneSingleIndexLanguageModel(Paths.get("/data/ngrams/wrong-dir"));
// after
LanguageModel lm = new LuceneSingleIndexLanguageModel(Paths.get("/data/ngrams/en")); // contains 1grams..3grams

Defensive patterns

Strategy: validation

Validate before calling

File dir = Paths.get("/data/ngrams/en").toFile();
if (dir == null || !new File(dir, "1grams").isDirectory())
  throw new IllegalStateException("Not a LanguageTool ngram index: " + dir);

Prevention

Use only official LanguageTool ngram index downloads matching your LT version
Verify archive checksums after download
Never point the model at a general-purpose Lucene index

When it happens

Trigger: Calling getTotalTokenCount() on a LuceneSingleIndexLanguageModel whose underlying index directory contains >1000 documents with a 'totalTokenCount' field, e.g. a wrong or corrupted 1grams index directory, or an index built by a different/incompatible FrequencyIndexCreator.

Common situations: Pointing ngram.languageModel at the wrong directory (top-level dir instead of the per-ngrams dirs), using an index built by a different Lucene LanguageTool version, or a partially copied/downloaded index that got mixed with other documents.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/42ec274426559d56. Report an issue: GitHub.