languagetool-org/languagetool · error

Got more than one hit for:

Error message

Got more than one hit for: 

What it means

CommonCrawlToNgram.writeToLucene() expects each ngram to correspond to at most one document in the Lucene index. After fetching topDocs for the ngram, if totalHits > 1 the index holds duplicate entries for the same ngram, so the delete-and-update logic (which assumes a single prior doc) cannot determine the correct old count and the tool throws.

Source

Thrown at languagetool-dev/src/main/java/org/languagetool/dev/bigdata/CommonCrawlToNgram.java:199

    index.searcher = new IndexSearcher(index.reader);
    for (Map.Entry<String, Long> entry : ngramToCount.entrySet()) {
      Term ngram = new Term("ngram", entry.getKey());
      TopDocs topDocs = index.searcher.search(new TermQuery(ngram), 2);
      //System.out.println(ngram + " ==> " + topDocs.totalHits);
      if (topDocs.totalHits == 0) {
        Document doc = getDoc(entry.getKey(), entry.getValue());
        index.indexWriter.addDocument(doc);
      } else if (topDocs.totalHits == 1) {
        int docNumber = topDocs.scoreDocs[0].doc;
        Document document = index.reader.document(docNumber);
        long oldCount = Long.parseLong(document.getField("count").stringValue());
        //System.out.println(ngram + " -> " + oldCount + "+" + entry.getValue());
        index.indexWriter.deleteDocuments(ngram);
        index.indexWriter.addDocument(getDoc(entry.getKey(), oldCount + entry.getValue()));
        // would probably be faster, but we currently rely on the count being a common field:
        //indexWriter.updateNumericDocValue(ngram, "count", oldCount + entry.getValue());
      } else if (topDocs.totalHits > 1) {
        throw new RuntimeException("Got more than one hit for: " + ngram);
      }
      //System.out.println("   " + entry.getKey() + " -> " + entry.getValue());
    }
    if (ngramSize == 1) {
      // TODO: runtime code will crash if there are more than 1000 of these docs, so update instead of delete
      long total = ngramToCount.values().stream().mapToLong(Number::longValue).sum();
      System.out.println("Adding totalTokenCount doc: " + total);
      addTotalTokenCountDoc(total, index.indexWriter);
    }
    System.out.println("Commit...");
    index.indexWriter.commit();
    System.out.println("Commit done, indexing took " + (System.currentTimeMillis()-startTime) + "ms");
    ngramToCount.clear();
  }

  @NotNull
  private Document getDoc(String ngram, long count) {
    Document doc = new Document();

View on GitHub (pinned to 2e990059ce)

Solutions

1. Rebuild the index from scratch in a clean directory so every ngram maps to exactly one document.
2. Deduplicate existing indexes: for each duplicate ngram, sum counts, deleteDocuments(ngram), and re-add a single doc.
3. Ensure every write path deletes the ngram before adding (the code already does deleteDocuments before addDocument on the single-hit path — apply the same on rebuild).
4. Consider using a single indexed unique field or updateDocument/atomic replace semantics to prevent duplicates structurally.

Example fix

// before
} else if (topDocs.totalHits > 1) {
  throw new RuntimeException("Got more than one hit for: " + ngram);
}
// after
} else if (topDocs.totalHits > 1) {
  long summed = 0;
  for (ScoreDoc sd : topDocs.scoreDocs) {
    summed += Long.parseLong(reader.document(sd.doc).get("count"));
  }
  index.indexWriter.deleteDocuments(ngram);
  index.indexWriter.addDocument(getDoc(ngram, summed));
}

// check uniqueness of ngram docs before incremental updates
TopDocs td = searcher.search(new TermQuery(new Term("ngram", ngram)), 2);
if (td.totalHits > 1) {
  throw new IllegalStateException("Duplicate ngram docs in index: " + ngram);
}

try {
  writer.writeAndEvaluate(...);
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("Got more than one hit")) {
    rebuildIndexInCleanDir();
  }
}

When it happens

Trigger: Updating counts for an ngram whose term query matches multiple documents — usually after the index was built more than once into the same directory (docs appended without deletion), or when duplicate keys were indexed from overlapping input files.

Common situations: Re-running the ngram indexing pipeline over an existing index directory; merging partial indexes with overlapping ngrams; interrupted runs that left partial duplicate documents behind.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• More hits than expected for
• No hits for
• No ngram data found for:
• Directory must contain at least '1grams', '2grams', and '3gr
• Expected at least '1grams', '2grams', and '3grams' sub direc

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/96dee3a55cc0a6f4. Report an issue: GitHub.