stanfordnlp/CoreNLP · error · RuntimeException

gradient check failed

Error message

gradient check failed

What it means

During non-linear CRF training (trainWeightsUsingNonLinearCRF, invoked from allWeights), when flags.checkGradient is enabled the numeric gradient of the objective is verified against the analytic gradient via func.gradientCheck(). If the check fails, this RuntimeException is thrown before optimization starts. It means the implemented gradient does not match finite-difference approximations, usually indicating a bug in the feature/gradient computation or a numerical issue (NaN/Inf values, bad scaling).

Solutions

1. Inspect the training data and features for NaN, Inf, or empty feature vectors; remove or fix offending examples.
2. Ensure every weight index entry is produced by a feature extractor so value and gradient use identical feature sets.
3. If the failure stems from numerical precision, relax the check or check with a smaller test dataset / double precision.
4. If you are not debugging the objective, disable -checkGradient to proceed with training.
5. If a code change introduced the bug, revert or correct the gradient implementation so it matches the objective value computation.

Example fix

// before
trainClassifier props: -useNonLinearCRF true -checkGradient true  // throws if gradient is wrong
// after
trainClassifier props: -useNonLinearCRF true -checkGradient true -featureFactory CustomFeatureFactory // fix the extractor, keep checkGradient as a regression test

// Before training, sanitize data/features so the gradient check can pass:
for (List<IN> doc : trainingData) {
  for (CoreLabel w : doc) {
    if (Double.isNaN(doc.hashCode()) || w.get(CoreAnnotations.TextAnnotation.class).isEmpty()) {
      throw new IllegalArgumentException("Empty/invalid token in training data: " + w);
    }
  }
}
// only pass -checkGradient when debugging the objective itself

try {
  classifier.train(trainingData);
} catch (RuntimeException e) {
  if ("gradient check failed".equals(e.getMessage())) {
    log.error("Analytic gradient mismatch: verify feature extractor and objective before training.");
    // retrain with checkGradient disabled only after fixing the extractor
  } else throw e;
}

When it happens

Trigger: Running CRF training with -checkGradient (flags.checkGradient=true) on a CRFClassifierNonlinear model when the CGTester/NonlinearCRAFunctional gradientCheck() returns false: analytic gradient disagrees with finite-difference gradient beyond tolerance, e.g. due to NaN weights, duplicated/missing features between weight indices and the function, or mismatched featureIndex contents.

Common situations: Developers enabling gradient checking to debug a custom feature or objective; training data or features producing extreme/NaN values; modifying the CRF objective code so the gradient implementation drifts from the value function; running on data where log-likelihood underflows.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Could not read from double initial LOP weights file
• Could not read from double initial LOP scales file
• CRFLogConditionalObjectiveFloatFunction is not clique potent
• Unknown prior type:
• Got NaN for prob in CRFLogConditionalObjectiveFunction.calcu

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/8d8c7c7c337168af. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/ie/crf/CRFClassifierNonlinear.java:176

    log.info("numWeights: " + initialWeights.length);

    if (flags.testObjFunction) {
      StochasticDiffFunctionTester tester = new StochasticDiffFunctionTester(func);
      if (tester.testSumOfBatches(initialWeights, 1e-4)) {
        log.info("Testing complete... exiting");
        System.exit(1);
      } else {
        log.info("Testing failed....exiting");
        System.exit(1);
      }

    }
    //check gradient
    if (flags.checkGradient) {
      if (func.gradientCheck()) {
        log.info("gradient check passed");
      } else {
        throw new RuntimeException("gradient check failed");
      }
    }
    return minimizer.minimize(func, flags.tolerance, initialWeights);
  }

  @Override
  protected void serializeTextClassifier(PrintWriter pw) throws Exception {
    super.serializeTextClassifier(pw);

    pw.printf("nodeFeatureIndicesMap.size()=\t%d%n", nodeFeatureIndicesMap.size());
    for (int i = 0; i < nodeFeatureIndicesMap.size(); i++) {
      pw.printf("%d\t%d%n", i, nodeFeatureIndicesMap.get(i));
    }

    pw.printf("edgeFeatureIndicesMap.size()=\t%d%n", edgeFeatureIndicesMap.size());
    for (int i = 0; i < edgeFeatureIndicesMap.size(); i++) {
      pw.printf("%d\t%d%n", i, edgeFeatureIndicesMap.get(i));
    }

View on GitHub (pinned to 1b7edd19c4)