{"record":{"id":"5349a19d62ff3e7a","repo":"stanfordnlp/CoreNLP","slug":"tried-to-compare-two-distribution-k-objects-but-d-5349a1","errorCode":null,"errorMessage":"Tried to compare two Distribution<K> objects but d1.counter intersect d2.counter > numberOfKeys","messagePattern":"Tried to compare two Distribution<K> objects but d1\\.counter intersect d2\\.counter > numberOfKeys","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/stats/Distributions.java","lineNumber":32,"sourceCode":" *\n *\n * @author Jeff Michels (jmichels@stanford.edu)\n */\npublic class Distributions {\n\n  private Distributions() {\n  }\n\n\n  protected static <K> Set<K> getSetOfAllKeys(Distribution<K> d1, Distribution<K> d2) {\n    if (d1.getNumberOfKeys() != d2.getNumberOfKeys()){\n      throw new RuntimeException(\"Tried to compare two Distribution<K> objects but d1.numberOfKeys != d2.numberOfKeys\");\n    }\n\n    Set<K> allKeys = Generics.newHashSet(d1.getCounter().keySet());\n    allKeys.addAll(d2.getCounter().keySet());\n    if (allKeys.size() > d1.getNumberOfKeys()){\n      throw new RuntimeException(\"Tried to compare two Distribution<K> objects but d1.counter intersect d2.counter > numberOfKeys\");\n    }\n    return allKeys;\n  }\n\n  /**\n   * Returns a double between 0 and 1 representing the overlap of d1 and d2.\n   * Equals 0 if there is no overlap, equals 1 iff d1==d2\n   */\n  public static <K> double overlap(Distribution<K> d1, Distribution<K> d2) {\n    Set<K> allKeys = getSetOfAllKeys(d1, d2);\n\n    double result = 0.0;\n    double remainingMass1 = 1.0;\n    double remainingMass2 = 1.0;\n\n    for (K key : allKeys){\n      double p1 = d1.probabilityOf(key);\n      double p2 = d2.probabilityOf(key);","sourceCodeStart":14,"sourceCodeEnd":50,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/stats/Distributions.java#L14-L50","documentation":"After taking the union of the two distributions' key sets, getSetOfAllKeys() checks an invariant: the union size must not exceed d1's numberOfKeys, because all keys of both distributions must belong to the shared universe of that size. A larger union proves the distributions were built with inconsistent numberOfKeys or contain keys outside the declared universe, so a RuntimeException is thrown (the message says 'intersect' but the check is on the union).","triggerScenarios":"Comparing two distributions where the combined distinct keys exceed d1.getNumberOfKeys() — e.g. numberOfKeys was under-declared, or the distributions were built over genuinely different key spaces with the same declared count.","commonSituations":"Under-reporting numberOfKeys when constructing distributions manually (Distributionconstructor with explicit numberOfKeys) then comparing via Distributions.overlap / intersection / jaccardCoefficient.","solutions":["Increase numberOfKeys on both distributions to at least the size of the true combined key universe","Rebuild both distributions consistently from the full key space","Compare via your own union code if the invariant of shared numberOfKeys does not hold for your data"],"exampleFix":"// before\nDistribution<String> d1 = new Distribution<>(c1, 10); // too small\nDistribution<String> d2 = new Distribution<>(c2, 10);\ndouble j = Distributions.jaccardCoefficient(d1, d2); // union of keys > 10\n// after\nSet<String> universe = union(c1.keySet(), c2.keySet());\nDistribution<String> d1 = new Distribution<>(c1, universe.size());\nDistribution<String> d2 = new Distribution<>(c2, universe.size());","handlingStrategy":"validation","validationCode":"Set<K> union = new HashSet<>(d1.getCounter().keySet());\nunion.addAll(d2.getCounter().keySet());\nif (union.size() > d1.getNumberOfKeys()) {\n  throw new IllegalStateException(\"union of keys exceeds numberOfKeys=\" + d1.getNumberOfKeys());\n}","typeGuard":null,"tryCatchPattern":"try {\n  double overlap = Distributions.overlap(d1, d2);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"intersect d2.counter\")) { /* fix numberOfKeys declarations */ }\n}","preventionTips":["Set numberOfKeys >= size of the combined key universe","Avoid hard-coded numberOfKeys values; derive from data","Test comparisons with distributions built from overlapping-but-different vocabularies"],"tags":["java","distribution-comparison","invariant","statistics"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}