languagetool-org/languagetool · error · RuntimeException

Expected 'totalTokenCount' meta documents not found in 1gram

Error message

Expected 'totalTokenCount' meta documents not found in 1grams index: ${luceneSearcher.directory}

What it means

getTotalTokenCount() scans the 1grams index for special meta documents carrying the 'totalTokenCount' field to compute the corpus size. If the regexp query finds zero such documents, it throws RuntimeException naming the index directory — the index lacks the required metadata.

Source

Thrown at languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java:150

  @Override
  public long getCount(String token1) {
    Objects.requireNonNull(token1);
    //TODO: move this into the document? It's not there currently...
    //if (token1.equals(LanguageModel.GOOGLE_SENTENCE_START)) {
    //  return 42_107_029_039L;  // see StartTokenCounter, run with 2grams (3grams: 124_541_229_392)
    //}
    return getCount(Arrays.asList(token1));
  }

  @Override
  public long getTotalTokenCount() {
    LuceneSearcher luceneSearcher = getLuceneSearcher(1);
    try {
      RegexpQuery query = new RegexpQuery(new Term("totalTokenCount", ".*"));
      TopDocs docs = luceneSearcher.searcher.search(query, 1000);  // Integer.MAX_VALUE might cause OOE on wrong index
      if (docs.totalHits == 0) {
        throw new RuntimeException("Expected 'totalTokenCount' meta documents not found in 1grams index: " + luceneSearcher.directory);
      } else if (docs.totalHits > 1000) {
        throw new RuntimeException("Did not expect more than 1000 'totalTokenCount' meta documents: " + docs.totalHits + " in " + luceneSearcher.directory);
      } else {
        long result = 0;
        for (ScoreDoc scoreDoc : docs.scoreDocs) {
          long tmp = Long.parseLong(luceneSearcher.reader.document(scoreDoc.doc).get("totalTokenCount"));
          if (tmp > result) {
            // due to the way FrequencyIndexCreator adds these totalTokenCount fields, we must not sum them,
            // but take the largest one:
            result = tmp;
          }
        }
        return result;
      }
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }

View on GitHub (pinned to 2e990059ce)

Solutions

Rebuild the 1grams index with LanguageTool's index builder so totalTokenCount meta documents are included
Download the official LanguageTool ngram index for the language
Compute the token count yourself and avoid getTotalTokenCount() for custom indexes

Example fix

// before
long total = lm.getTotalTokenCount(); // throws on custom index
// after
long total = hasTotalTokenCountMeta(indexDir) ? lm.getTotalTokenCount() : countTokensManually(indexDir);

Defensive patterns

Strategy: try-catch

Validate before calling

try (IndexReader r = DirectoryReader.open(FSDirectory.open(oneGramsDir))) {
  long meta = new IndexSearcher(r).count(new RegexpQuery(new Term("totalTokenCount", ".*")));
  if (meta == 0) throw new IllegalStateException("1grams index lacks totalTokenCount meta docs");
}

Try / catch

try { long total = lm.getTotalTokenCount(); } catch (RuntimeException e) { if (e.getMessage().startsWith("Expected 'totalTokenCount'")) { total = estimateTokenCount(lm); } else throw e; }

Prevention

Only use indexes produced by LanguageTool's official ngram indexing tooling
Verify index metadata (totalTokenCount docs) after download or rebuild
Cache the total token count instead of re-querying Lucene repeatedly

When it happens

Trigger: Calling getTotalTokenCount() on a 1grams Lucene index that was built without totalTokenCount meta documents (custom/self-built or older index).

Common situations: Using hand-built or third-party ngram indexes not created by LanguageTool's indexing tooling; upgraded model code expecting metadata missing in old indexes.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/d2dee2e7168ae7b1. Report an issue: GitHub.