stanfordnlp/CoreNLP · error · RuntimeException

OOPS... no prior distribution...?

Error message

OOPS... no prior distribution...?

What it means

getBackedOffDist walks the character-context hierarchy from the longest context down to the empty (prior) context looking up charDistributions. If even the empty context is missing, the smoothing invariant is broken and a RuntimeException is thrown.

Solutions

1. Retrain the lexicon on a non-empty, valid treebank so the prior distribution is built by finishTraining()
2. Reload the lexicon from a known-good serialized file rather than a possibly corrupt one
3. Patch getBackedOffDist to return a uniform/minimum distribution when the lookup misses, instead of throwing
4. Check that finishTraining() was called before scoring

Example fix

// before
throw new RuntimeException("OOPS... no prior distribution...?");
// after (defensive fallback)
Distribution<Character> prior = charDistributions.get(Collections.emptyList());
if (prior == null) { prior = Distribution.uniform(...); }
return prior;

// ensure the lexicon was trained before scoring
if (charDistributions == null || charDistributions.isEmpty())
  throw new IllegalStateException("Lexicon not trained: call finishTraining() first");

try { dist = lex.getBackedOffDist(context); } catch (RuntimeException e) { dist = uniformPrior; log.error("Missing prior distribution — retrain or reload lexicon", e); }

When it happens

Trigger: Calling the lexicon's scoring/sampling paths (charScore, d, sampleFrom) after finishTraining() when the prior (empty-context) distribution was never stored in charDistributions — e.g., training data produced no prior, or the lexicon was used before/without proper training/serialization round-trip.

Common situations: Loading a corrupted or truncated serialized Chinese character lexicon, training on an empty or degenerate tree set, custom code clearing or rebuilding charDistributions.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• BinaryHeadFinder: unexpected tree:
• ChineseCharacterBasedLexicon has no rule iterator!
• Comparing a mention with itself for representativeness
• Unknown minimizer
• : Parser grammar does not exist

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/e85ca9e728a1250a. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/parser/lexparser/ChineseCharacterBasedLexicon.java:303

        score -= (chars.length - 1) * lengthPenalty;
        break;
    }
    return (float) score;
  }


  // this is where we do backing off for unseen contexts
  // (backing off for rarely seen contexts is done implicitly
  // because the distributions are smoothed)
  private Distribution<Symbol> getBackedOffDist(List<Serializable> context) {
    // context contains [tag prevChar prevPrevChar]
    for (int i = CONTEXT_LENGTH + 1; i >= 0; i--) {
      List<Serializable> l = context.subList(0, i);
      if (charDistributions.containsKey(l)) {
        return charDistributions.get(l);
      }
    }
    throw new RuntimeException("OOPS... no prior distribution...?");
  }

  /**
   * Samples from the distribution over words with this POS according to the lexicon.
   *
   * @param tag the POS of the word to sample
   * @return a sampled word
   */
  public String sampleFrom(String tag) {
    StringBuilder buf = new StringBuilder();
    List<Serializable> context = new ArrayList<>(CONTEXT_LENGTH + 1);

    // context must contain [tag prevChar prevPrevChar]
    context.add(tag);
    for (int i = 0; i < CONTEXT_LENGTH; i++) {
      context.add(Symbol.BEGIN_WORD);
    }
    Distribution<Symbol> d = getBackedOffDist(context);

View on GitHub (pinned to 1b7edd19c4)