What causes "ERROR: the probability distribution sums to ${sum}" in stanfordnlp/CoreNLP?

Inspect r/n arrays for degenerate or pathological values (e.g., all counts identical) that break the regression and adjust or clean the data.

stanfordnlp/CoreNLP · error · IllegalStateException

ERROR: the probability distribution sums to

Error message

ERROR: the probability distribution sums to ${sum}

What it means

After computing the Good-Turing smoothed probabilities, validate() checks that pZero plus the sum of n[i]*p[i] over all bins is within `tolerance` of 1.0, since the result must be a probability distribution. If the computed distribution deviates more than the tolerance, the internal numerical fit failed and IllegalStateException is thrown.

Solutions

Inspect r/n arrays for degenerate or pathological values (e.g., all counts identical) that break the regression and adjust or clean the data.
Verify the input arrays accurately represent frequency-of-frequency counts from your corpus; recomputing from raw data often fixes the sum.
If the deviation is small and acceptable for your use case, patch or vendor the class to relax `tolerance`.

Example fix

// before
SimpleGoodTuring sgt = new SimpleGoodTuring(r, n); // may throw IllegalStateException
// after
double[] probs;
try {
  SimpleGoodTuring sgt = new SimpleGoodTuring(r, n);
  probs = sgt.getProbabilities();
} catch (IllegalStateException e) {
  probs = fallbackSimpleSmoothing(r, n); // e.g. add-one smoothing
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Sanity-check frequency data before construction:
double total = 0;
for (int i = 0; i < r.length; i++) total += (double) r[i] * n[i];
boolean sane = r.length >= 5 && total > 0;

Try / catch

// Java
try {
  SimpleGoodTuring sgt = new SimpleGoodTuring(r, n);
  double[] p = sgt.getProbabilities();
} catch (IllegalStateException e) {
  double[] p = simpleAddOneSmoothing(r, n); // deterministic fallback
}

Prevention

Ensure r/n arrays are exact frequency-of-frequency counts derived from the corpus.
Avoid degenerate inputs (e.g., all counts identical).
Wrap model construction in try-catch with a simpler smoothing fallback.

When it happens

Trigger: Calling new SimpleGoodTuring(r, n) (which runs compute() then validate()) with input data for which the linear regression on log frequency vs log frequency-of-frequency produces probabilities that don't sum to 1 within tolerance.

Common situations: Highly skewed or pathological frequency distributions, degenerate inputs where the SGT regression fit breaks down, or data containing extreme frequency counts that destabilize the smoothing computation.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/13201d001cbbade0. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/stats/SimpleGoodTuring.java:183

    int i;
    System.out.printf("%6s %6s %8s %8s%n", "r", "n", "p", "p*");
    System.out.printf("%6s %6s %8s %8s%n", "----", "----", "----", "----");
    System.out.printf("%6d %6d %8.4g %8.4g%n", 0, 0, 0.0, pZero);
    for (i = 0; i < rows; ++i)
      System.out.printf("%6d %6d %8.4g %8.4g%n", r[i], n[i], 1.0 * r[i] / bigN, p[i]);
  }

  /**
   * Ensures that we have a proper probability distribution.
   */
  private void validate(double tolerance) {
    double sum = pZero;
    for (int i = 0; i < n.length; i++) {
      sum += (n[i] * p[i]);
    }
    double err = 1.0 - sum;
    if (Math.abs(err) > tolerance) {
      throw new IllegalStateException("ERROR: the probability distribution sums to " + sum);
    }
  }


  // static methods -------------------------------------------------------------

  /**
   * Reads from STDIN a sequence of lines, each containing two integers,
   * separated by whitespace.  Returns a pair of int arrays containing the
   * values read.
   */
  private static int[][] readInput() throws Exception {
    List<Integer> rVals = new ArrayList<>();
    List<Integer> nVals = new ArrayList<>();
    BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
    String line;
    while ((line = in.readLine()) != null) {
      String[] tokens = line.trim().split("\\s+");

View on GitHub (pinned to 1b7edd19c4)