{"record":{"id":"6d8a8014eddf7178","repo":"languagetool-org/languagetool","slug":"more-hits-than-expected-for","errorCode":null,"errorMessage":"More hits than expected for ","messagePattern":"More hits than expected for ","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"languagetool-dev/src/main/java/org/languagetool/dev/archive/StartTokenCounter.java","lineNumber":75,"sourceCode":"        if (term.startsWith(LanguageModel.GOOGLE_SENTENCE_START)) {\n          if (term.matches(\".*_(ADJ|ADV|NUM|VERB|ADP|NOUN|PRON|CONJ|DET|PRT)$\")) {\n            //System.out.println(\"ignore: \" + term);\n            continue;\n          }\n          TopDocs topDocs = searcher.search(new TermQuery(new Term(\"ngram\", term)), 3);\n          if (topDocs.totalHits == 0) {\n            throw new RuntimeException(\"No hits for \" + term + \": \" + topDocs.totalHits);\n          } else if (topDocs.totalHits == 1) {\n            int docId = topDocs.scoreDocs[0].doc;\n            Document document = reader.document(docId);\n            Long count = Long.parseLong(document.get(\"count\"));\n            //System.out.println(term + \" -> \" + count);\n            totalCount += count;\n            if (++i % 10_000 == 0) {\n              System.out.println(i + \" ... \" + totalCount);\n            }\n          } else {\n            throw new RuntimeException(\"More hits than expected for \" + term + \": \" + topDocs.totalHits);\n          }\n        }\n      }\n    }\n    System.out.println(\"==> \" + totalCount);\n  }\n  \n}\n","sourceCodeStart":57,"sourceCodeEnd":84,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-dev/src/main/java/org/languagetool/dev/archive/StartTokenCounter.java#L57-L84","documentation":"In StartTokenCounter, each ngram term is expected to match at most one document in the Lucene index (topHits limited to 3). If topDocs.totalHits > 1, the index contains duplicate documents for the same ngram, making the count ambiguous, so the tool throws 'More hits than expected'.","triggerScenarios":"The Lucene index contains more than one document with the same \"ngram\" field value — typically because the index was built multiple times without cleaning (docs appended instead of replaced), or the ngram field was not indexed as a unique key.","commonSituations":"Re-running AggregatedNgramToLucene against a non-empty index directory, duplicating every entry; merging indexes from multiple corpus files with overlapping ngrams; a corrupted or partially deleted index where deleteDocuments wasn't applied.","solutions":["Rebuild the index in a clean/empty directory so each ngram exists exactly once.","If duplicates exist, deduplicate: delete all docs for the term and re-add with the summed count before counting.","Harden CommonCrawlToNgram/AggregatedNgramToLucene to always deleteDocuments(ngram) before addDocument so rebuilds don't duplicate.","As a workaround in StartTokenCounter, sum counts over all hits instead of throwing."],"exampleFix":"// before\n} else {\n  throw new RuntimeException(\"More hits than expected for \" + term + \": \" + topDocs.totalHits);\n}\n// after\n} else {\n  long summed = 0;\n  for (ScoreDoc sd : topDocs.scoreDocs) {\n    summed += Long.parseLong(reader.document(sd.doc).get(\"count\"));\n  }\n  totalCount += summed;\n}","handlingStrategy":"validation","validationCode":"// detect duplicate ngram docs after building the index\nfor (String sample : sampledTerms) {\n  if (searcher.search(new TermQuery(new Term(\"ngram\", sample)), 5).totalHits > 1) {\n    throw new IllegalStateException(\"Duplicate docs for ngram: \" + sample);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  writeCounts(ngramToCount);\n} catch (RuntimeException e) {\n  if (e.getMessage().startsWith(\"More hits than expected\")) {\n    rebuildIndexClean(); // wipe directory and re-index\n  }\n}","preventionTips":["Always index into a clean/empty directory; delete or version index dirs between runs","Issue deleteDocuments(term) before every addDocument on update paths","After building an index, run a duplicate-detection sanity check","Never append to an existing ngram index with overlapping input"],"tags":["java","lucene","ngram","duplicate-data"],"backgroundTag":"internal-invariant-violation","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}