{"record":{"id":"e85ca9e728a1250a","repo":"stanfordnlp/CoreNLP","slug":"oops-no-prior-distribution","errorCode":null,"errorMessage":"OOPS... no prior distribution...?","messagePattern":"OOPS\\.\\.\\. no prior distribution\\.\\.\\.\\?","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/parser/lexparser/ChineseCharacterBasedLexicon.java","lineNumber":303,"sourceCode":"        score -= (chars.length - 1) * lengthPenalty;\n        break;\n    }\n    return (float) score;\n  }\n\n\n  // this is where we do backing off for unseen contexts\n  // (backing off for rarely seen contexts is done implicitly\n  // because the distributions are smoothed)\n  private Distribution<Symbol> getBackedOffDist(List<Serializable> context) {\n    // context contains [tag prevChar prevPrevChar]\n    for (int i = CONTEXT_LENGTH + 1; i >= 0; i--) {\n      List<Serializable> l = context.subList(0, i);\n      if (charDistributions.containsKey(l)) {\n        return charDistributions.get(l);\n      }\n    }\n    throw new RuntimeException(\"OOPS... no prior distribution...?\");\n  }\n\n  /**\n   * Samples from the distribution over words with this POS according to the lexicon.\n   *\n   * @param tag the POS of the word to sample\n   * @return a sampled word\n   */\n  public String sampleFrom(String tag) {\n    StringBuilder buf = new StringBuilder();\n    List<Serializable> context = new ArrayList<>(CONTEXT_LENGTH + 1);\n\n    // context must contain [tag prevChar prevPrevChar]\n    context.add(tag);\n    for (int i = 0; i < CONTEXT_LENGTH; i++) {\n      context.add(Symbol.BEGIN_WORD);\n    }\n    Distribution<Symbol> d = getBackedOffDist(context);","sourceCodeStart":285,"sourceCodeEnd":321,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/parser/lexparser/ChineseCharacterBasedLexicon.java#L285-L321","documentation":"getBackedOffDist walks the character-context hierarchy from the longest context down to the empty (prior) context looking up charDistributions. If even the empty context is missing, the smoothing invariant is broken and a RuntimeException is thrown.","triggerScenarios":"Calling the lexicon's scoring/sampling paths (charScore, d, sampleFrom) after finishTraining() when the prior (empty-context) distribution was never stored in charDistributions — e.g., training data produced no prior, or the lexicon was used before/without proper training/serialization round-trip.","commonSituations":"Loading a corrupted or truncated serialized Chinese character lexicon, training on an empty or degenerate tree set, custom code clearing or rebuilding charDistributions.","solutions":["Retrain the lexicon on a non-empty, valid treebank so the prior distribution is built by finishTraining()","Reload the lexicon from a known-good serialized file rather than a possibly corrupt one","Patch getBackedOffDist to return a uniform/minimum distribution when the lookup misses, instead of throwing","Check that finishTraining() was called before scoring"],"exampleFix":"// before\nthrow new RuntimeException(\"OOPS... no prior distribution...?\");\n// after (defensive fallback)\nDistribution<Character> prior = charDistributions.get(Collections.emptyList());\nif (prior == null) { prior = Distribution.uniform(...); }\nreturn prior;","handlingStrategy":"try-catch","validationCode":"// ensure the lexicon was trained before scoring\nif (charDistributions == null || charDistributions.isEmpty())\n  throw new IllegalStateException(\"Lexicon not trained: call finishTraining() first\");","typeGuard":null,"tryCatchPattern":"try { dist = lex.getBackedOffDist(context); } catch (RuntimeException e) { dist = uniformPrior; log.error(\"Missing prior distribution — retrain or reload lexicon\", e); }","preventionTips":["Always call finishTraining() before scoring/sampling","Train on a non-empty, representative treebank","Validate serialized lexicons load with a non-empty distribution table"],"tags":["parser","lexicon","smoothing","illegal-state"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}