{"record":{"id":"d2dee2e7168ae7b1","repo":"languagetool-org/languagetool","slug":"expected-totaltokencount-meta-documents-not-foun","errorCode":null,"errorMessage":"Expected 'totalTokenCount' meta documents not found in 1grams index: ${luceneSearcher.directory}","messagePattern":"Expected 'totalTokenCount' meta documents not found in 1grams index: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java","lineNumber":150,"sourceCode":"\n  @Override\n  public long getCount(String token1) {\n    Objects.requireNonNull(token1);\n    //TODO: move this into the document? It's not there currently...\n    //if (token1.equals(LanguageModel.GOOGLE_SENTENCE_START)) {\n    //  return 42_107_029_039L;  // see StartTokenCounter, run with 2grams (3grams: 124_541_229_392)\n    //}\n    return getCount(Arrays.asList(token1));\n  }\n\n  @Override\n  public long getTotalTokenCount() {\n    LuceneSearcher luceneSearcher = getLuceneSearcher(1);\n    try {\n      RegexpQuery query = new RegexpQuery(new Term(\"totalTokenCount\", \".*\"));\n      TopDocs docs = luceneSearcher.searcher.search(query, 1000);  // Integer.MAX_VALUE might cause OOE on wrong index\n      if (docs.totalHits == 0) {\n        throw new RuntimeException(\"Expected 'totalTokenCount' meta documents not found in 1grams index: \" + luceneSearcher.directory);\n      } else if (docs.totalHits > 1000) {\n        throw new RuntimeException(\"Did not expect more than 1000 'totalTokenCount' meta documents: \" + docs.totalHits + \" in \" + luceneSearcher.directory);\n      } else {\n        long result = 0;\n        for (ScoreDoc scoreDoc : docs.scoreDocs) {\n          long tmp = Long.parseLong(luceneSearcher.reader.document(scoreDoc.doc).get(\"totalTokenCount\"));\n          if (tmp > result) {\n            // due to the way FrequencyIndexCreator adds these totalTokenCount fields, we must not sum them,\n            // but take the largest one:\n            result = tmp;\n          }\n        }\n        return result;\n      }\n    } catch (IOException e) {\n      throw new RuntimeException(e);\n    }\n  }","sourceCodeStart":132,"sourceCodeEnd":168,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-core/src/main/java/org/languagetool/languagemodel/LuceneSingleIndexLanguageModel.java#L132-L168","documentation":"getTotalTokenCount() scans the 1grams index for special meta documents carrying the 'totalTokenCount' field to compute the corpus size. If the regexp query finds zero such documents, it throws RuntimeException naming the index directory — the index lacks the required metadata.","triggerScenarios":"Calling getTotalTokenCount() on a 1grams Lucene index that was built without totalTokenCount meta documents (custom/self-built or older index).","commonSituations":"Using hand-built or third-party ngram indexes not created by LanguageTool's indexing tooling; upgraded model code expecting metadata missing in old indexes.","solutions":["Rebuild the 1grams index with LanguageTool's index builder so totalTokenCount meta documents are included","Download the official LanguageTool ngram index for the language","Compute the token count yourself and avoid getTotalTokenCount() for custom indexes"],"exampleFix":"// before\nlong total = lm.getTotalTokenCount(); // throws on custom index\n// after\nlong total = hasTotalTokenCountMeta(indexDir) ? lm.getTotalTokenCount() : countTokensManually(indexDir);","handlingStrategy":"try-catch","validationCode":"try (IndexReader r = DirectoryReader.open(FSDirectory.open(oneGramsDir))) {\n  long meta = new IndexSearcher(r).count(new RegexpQuery(new Term(\"totalTokenCount\", \".*\")));\n  if (meta == 0) throw new IllegalStateException(\"1grams index lacks totalTokenCount meta docs\");\n}","typeGuard":null,"tryCatchPattern":"try { long total = lm.getTotalTokenCount(); } catch (RuntimeException e) { if (e.getMessage().startsWith(\"Expected 'totalTokenCount'\")) { total = estimateTokenCount(lm); } else throw e; }","preventionTips":["Only use indexes produced by LanguageTool's official ngram indexing tooling","Verify index metadata (totalTokenCount docs) after download or rebuild","Cache the total token count instead of re-querying Lucene repeatedly"],"tags":["java","lucene","ngrams","index-metadata"],"backgroundTag":"missing-required-config-field","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}