What causes "More hits than expected for " in languagetool-org/languagetool?

Rebuild the index in a clean/empty directory so each ngram exists exactly once.

More hits than expected for

Error message

More hits than expected for

What it means

In StartTokenCounter, each ngram term is expected to match at most one document in the Lucene index (topHits limited to 3). If topDocs.totalHits > 1, the index contains duplicate documents for the same ngram, making the count ambiguous, so the tool throws 'More hits than expected'.

Source

Thrown at languagetool-dev/src/main/java/org/languagetool/dev/archive/StartTokenCounter.java:75

        if (term.startsWith(LanguageModel.GOOGLE_SENTENCE_START)) {
          if (term.matches(".*_(ADJ|ADV|NUM|VERB|ADP|NOUN|PRON|CONJ|DET|PRT)$")) {
            //System.out.println("ignore: " + term);
            continue;
          }
          TopDocs topDocs = searcher.search(new TermQuery(new Term("ngram", term)), 3);
          if (topDocs.totalHits == 0) {
            throw new RuntimeException("No hits for " + term + ": " + topDocs.totalHits);
          } else if (topDocs.totalHits == 1) {
            int docId = topDocs.scoreDocs[0].doc;
            Document document = reader.document(docId);
            Long count = Long.parseLong(document.get("count"));
            //System.out.println(term + " -> " + count);
            totalCount += count;
            if (++i % 10_000 == 0) {
              System.out.println(i + " ... " + totalCount);
            }
          } else {
            throw new RuntimeException("More hits than expected for " + term + ": " + topDocs.totalHits);
          }
        }
      }
    }
    System.out.println("==> " + totalCount);
  }
  
}

View on GitHub (pinned to 2e990059ce)

Solutions

Rebuild the index in a clean/empty directory so each ngram exists exactly once.
If duplicates exist, deduplicate: delete all docs for the term and re-add with the summed count before counting.
Harden CommonCrawlToNgram/AggregatedNgramToLucene to always deleteDocuments(ngram) before addDocument so rebuilds don't duplicate.
As a workaround in StartTokenCounter, sum counts over all hits instead of throwing.

Example fix

// before
} else {
  throw new RuntimeException("More hits than expected for " + term + ": " + topDocs.totalHits);
}
// after
} else {
  long summed = 0;
  for (ScoreDoc sd : topDocs.scoreDocs) {
    summed += Long.parseLong(reader.document(sd.doc).get("count"));
  }
  totalCount += summed;
}

Defensive patterns

Strategy: validation

Validate before calling

// detect duplicate ngram docs after building the index
for (String sample : sampledTerms) {
  if (searcher.search(new TermQuery(new Term("ngram", sample)), 5).totalHits > 1) {
    throw new IllegalStateException("Duplicate docs for ngram: " + sample);
  }
}

Try / catch

try {
  writeCounts(ngramToCount);
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("More hits than expected")) {
    rebuildIndexClean(); // wipe directory and re-index
  }
}

Prevention

Always index into a clean/empty directory; delete or version index dirs between runs
Issue deleteDocuments(term) before every addDocument on update paths
After building an index, run a duplicate-detection sanity check
Never append to an existing ngram index with overlapping input

When it happens

Trigger: The Lucene index contains more than one document with the same "ngram" field value — typically because the index was built multiple times without cleaning (docs appended instead of replaced), or the ngram field was not indexed as a unique key.

Common situations: Re-running AggregatedNgramToLucene against a non-empty index directory, duplicating every entry; merging indexes from multiple corpus files with overlapping ngrams; a corrupted or partially deleted index where deleteDocuments wasn't applied.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/6d8a8014eddf7178. Report an issue: GitHub.