{"record":{"id":"c1a7586af85ec814","repo":"stanfordnlp/CoreNLP","slug":"got-nan-for-prob-in-crflogconditionalobjectivefunc-c1a758","errorCode":null,"errorMessage":"Got NaN for prob in CRFLogConditionalObjectiveFunction.calculate()","messagePattern":"Got NaN for prob in CRFLogConditionalObjectiveFunction\\.calculate\\(\\)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/ie/crf/CRFLogConditionalObjectiveFunction.java","lineNumber":474,"sourceCode":"    return data.length;\n  }\n\n  @Override\n  public void calculateStochastic(double[] x, double [] v, int[] batch) {\n    to2D(x, weights);\n    setWeights(weights);\n\n    double batchScale = ((double) batch.length)/((double) this.dataDimension());\n\n    // the expectations over counts\n    // first index is feature index, second index is of possible labeling\n    // double[][] E = empty2D();\n\n    // iterate over all the documents\n    double prob = multiThreadGradient(batch, false);  // the log prob of the sequence given the model, which is the negation of value at this point\n\n    if (Double.isNaN(prob)) { // shouldn't be the case\n      throw new RuntimeException(\"Got NaN for prob in CRFLogConditionalObjectiveFunction.calculate()\");\n    }\n\n    value = -prob;\n\n    // compute the partial derivative for each feature by comparing expected counts to empirical counts\n    int index = 0;\n    for (int i = 0; i < E.length; i++) {\n      double[] E_i = E[i], Ehat_i = Ehat[i];\n      for (int j = 0; j < E_i.length; j++) {\n        // real gradient should be empirical-expected;\n        // but since we minimize -L(\\theta), the gradient is -(empirical-expected)\n        derivative[index++] = (E_i[j] - batchScale*Ehat_i[j]);\n        if (VERBOSE) {\n          log.info(\"deriv(\" + i + \",\" + j + \") = \" + E_i[j] + \" - \" + Ehat_i[j] + \" = \" + derivative[index - 1]);\n        }\n      }\n    }\n","sourceCodeStart":456,"sourceCodeEnd":492,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/ie/crf/CRFLogConditionalObjectiveFunction.java#L456-L492","documentation":"In the multi-threaded (batch) variant of CRFLogConditionalObjectiveFunction.calculate(), multiThreadGradient(batch, false) aggregates the per-document log-probabilities. If the aggregated prob is NaN the function throws, since a non-finite objective cannot be minimized. Unlike error 230 the message does not add the underflow hint, but the root cause is the same: NaN propagated from per-document gradient/value computation.","triggerScenarios":"Calling the batched calculate(...) overload (multi-threaded path) where one or more documents produce NaN log-probabilities or gradients (long sequences underflowing, NaN weights in x, or division by zero in feature values).","commonSituations":"Large-scale CRF training with multi-threading enabled, where a single bad document poisons the aggregated prob; also seen after a training step overshoots and weights become Inf/NaN.","solutions":["Validate the weight vector x for NaN/Inf at the start of each iteration and roll back or reduce the learning rate if found.","Split long documents into shorter sequences to avoid underflow.","Run with a single thread / smaller batch to isolate which document yields NaN.","Check feature values (featureVal) for zero or negative inputs used in log/exp computations.","Re-scale features and restart training from the last known-good model."],"exampleFix":"// before\nfor (double w : x) { /* no check */ }\ncrf.calculate(x, batch, E);\n// after\nfor (double w : x) {\n  if (Double.isNaN(w) || Double.isInfinite(w)) {\n    throw new IllegalStateException(\"bad weights; reduce learning rate\");\n  }\n}\ncrf.calculate(x, batch, E);","handlingStrategy":"try-catch","validationCode":"static boolean finiteParams(double[] x) { for (double w : x) if (!(Math.abs(w) < 1e10)) return false; return true; }\n// call before calculate: if (!finiteParams(x)) rollback();","typeGuard":"static boolean isFinite(double[] v) { for (double d : v) if (Double.isNaN(d) || Double.isInfinite(d)) return false; return true; }","tryCatchPattern":"try {\n  crf.calculate(x, batch, E);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"NaN for prob\")) {\n    batch = bisectBatch(batch); // isolate the offending document\n  } else throw e;\n}","preventionTips":["Pre-screen training documents for extreme length and split them.","Sanitize feature values (no NaN/Inf/negative where log/exp is used).","Checkpoint weights each iteration so you can roll back on NaN.","Run a smoke test on a small batch with multiple threads before the full run."],"tags":["crf","nan","multithreading","training"],"backgroundTag":"value-out-of-range","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}