{"record":{"id":"13201d001cbbade0","repo":"stanfordnlp/CoreNLP","slug":"error-the-probability-distribution-sums-to-sum","errorCode":null,"errorMessage":"ERROR: the probability distribution sums to ${sum}","messagePattern":"ERROR: the probability distribution sums to (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/stats/SimpleGoodTuring.java","lineNumber":183,"sourceCode":"    int i;\n    System.out.printf(\"%6s %6s %8s %8s%n\", \"r\", \"n\", \"p\", \"p*\");\n    System.out.printf(\"%6s %6s %8s %8s%n\", \"----\", \"----\", \"----\", \"----\");\n    System.out.printf(\"%6d %6d %8.4g %8.4g%n\", 0, 0, 0.0, pZero);\n    for (i = 0; i < rows; ++i)\n      System.out.printf(\"%6d %6d %8.4g %8.4g%n\", r[i], n[i], 1.0 * r[i] / bigN, p[i]);\n  }\n\n  /**\n   * Ensures that we have a proper probability distribution.\n   */\n  private void validate(double tolerance) {\n    double sum = pZero;\n    for (int i = 0; i < n.length; i++) {\n      sum += (n[i] * p[i]);\n    }\n    double err = 1.0 - sum;\n    if (Math.abs(err) > tolerance) {\n      throw new IllegalStateException(\"ERROR: the probability distribution sums to \" + sum);\n    }\n  }\n\n\n  // static methods -------------------------------------------------------------\n\n  /**\n   * Reads from STDIN a sequence of lines, each containing two integers,\n   * separated by whitespace.  Returns a pair of int arrays containing the\n   * values read.\n   */\n  private static int[][] readInput() throws Exception {\n    List<Integer> rVals = new ArrayList<>();\n    List<Integer> nVals = new ArrayList<>();\n    BufferedReader in = new BufferedReader(new InputStreamReader(System.in));\n    String line;\n    while ((line = in.readLine()) != null) {\n      String[] tokens = line.trim().split(\"\\\\s+\");","sourceCodeStart":165,"sourceCodeEnd":201,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/stats/SimpleGoodTuring.java#L165-L201","documentation":"After computing the Good-Turing smoothed probabilities, validate() checks that pZero plus the sum of n[i]*p[i] over all bins is within `tolerance` of 1.0, since the result must be a probability distribution. If the computed distribution deviates more than the tolerance, the internal numerical fit failed and IllegalStateException is thrown.","triggerScenarios":"Calling new SimpleGoodTuring(r, n) (which runs compute() then validate()) with input data for which the linear regression on log frequency vs log frequency-of-frequency produces probabilities that don't sum to 1 within tolerance.","commonSituations":"Highly skewed or pathological frequency distributions, degenerate inputs where the SGT regression fit breaks down, or data containing extreme frequency counts that destabilize the smoothing computation.","solutions":["Inspect r/n arrays for degenerate or pathological values (e.g., all counts identical) that break the regression and adjust or clean the data.","Verify the input arrays accurately represent frequency-of-frequency counts from your corpus; recomputing from raw data often fixes the sum.","If the deviation is small and acceptable for your use case, patch or vendor the class to relax `tolerance`."],"exampleFix":"// before\nSimpleGoodTuring sgt = new SimpleGoodTuring(r, n); // may throw IllegalStateException\n// after\ndouble[] probs;\ntry {\n  SimpleGoodTuring sgt = new SimpleGoodTuring(r, n);\n  probs = sgt.getProbabilities();\n} catch (IllegalStateException e) {\n  probs = fallbackSimpleSmoothing(r, n); // e.g. add-one smoothing\n}","handlingStrategy":"try-catch","validationCode":"// Sanity-check frequency data before construction:\ndouble total = 0;\nfor (int i = 0; i < r.length; i++) total += (double) r[i] * n[i];\nboolean sane = r.length >= 5 && total > 0;","typeGuard":null,"tryCatchPattern":"// Java\ntry {\n  SimpleGoodTuring sgt = new SimpleGoodTuring(r, n);\n  double[] p = sgt.getProbabilities();\n} catch (IllegalStateException e) {\n  double[] p = simpleAddOneSmoothing(r, n); // deterministic fallback\n}","preventionTips":["Ensure r/n arrays are exact frequency-of-frequency counts derived from the corpus.","Avoid degenerate inputs (e.g., all counts identical).","Wrap model construction in try-catch with a simpler smoothing fallback."],"tags":["nlp","smoothing","numerical"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}