What causes "gradient check failed" in stanfordnlp/CoreNLP?

Inspect training data for NaN/Inf or extreme feature values and clean/normalize them.

stanfordnlp/CoreNLP · error · RuntimeException

gradient check failed

Error message

gradient check failed

What it means

With flags.checkGradient=true, training calls func.gradientCheck() to numerically verify the analytic gradient of the CRF objective. A false result (analytic and finite-difference gradients disagree beyond tolerance) throws this RuntimeException. The mismatch usually indicates a broken derivative, bad features, or NaN/Inf values.

Solutions

Inspect training data for NaN/Inf or extreme feature values and clean/normalize them.
If you modified the objective or gradient code, fix the derivative implementation.
Re-run the check on a tiny model where the mismatch can be diagnosed numerically.
Remove checkGradient once validation passes — it is a debug flag.

Example fix

// before
props.setProperty("checkGradient", "true"); // throws on mismatch
// after (once the gradient is validated)
props.remove("checkGradient");

Defensive patterns

Strategy: try-catch

Validate before calling

boolean hasBadValues = features.stream().anyMatch(v -> Double.isNaN(v) || Double.isInfinite(v));
if ("true".equals(props.getProperty("checkGradient", "false")) && hasBadValues) {
  throw new IllegalArgumentException("NaN/Inf features will break the gradient check");
}

Try / catch

try {
  classifier.train(files);
} catch (RuntimeException e) {
  if ("gradient check failed".equals(e.getMessage())) {
    log.warn("Analytic vs numeric gradient mismatch; inspect objective/features");
    props.remove("checkGradient"); // after investigation
  } else throw e;
}

Prevention

Run gradient checks only on tiny models where failures are diagnosable.
Validate features for NaN/Inf and extreme magnitudes first.
Fix derivative code for any custom objective before proceeding.
Disable checkGradient for production runs.

When it happens

Trigger: Setting checkGradient=true and gradientCheck() returns false — commonly due to NaN/Inf in features or weights, extreme feature scaling, or a modified objective whose gradient is wrong.

Common situations: Debugging custom features or objective modifications; numerical instability with huge feature values; accidentally leaving the check enabled for large production runs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/179605fd7da24fef. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/ie/crf/CRFClassifier.java:1876

      }
    }
    log.info("numWeights: " + initialWeights.length);

    if (flags.testObjFunction) {
      StochasticDiffFunctionTester tester = new StochasticDiffFunctionTester(func);
      if (tester.testSumOfBatches(initialWeights, 1e-4)) {
        log.info("Successfully tested stochastic objective function.");
      } else {
        throw new IllegalStateException("Testing of stochastic objective function failed.");
      }

    }
    //check gradient
    if (flags.checkGradient) {
      if (func.gradientCheck()) {
        log.info("gradient check passed");
      } else {
        throw new RuntimeException("gradient check failed");
      }
    }
    return minimizer.minimize(func, flags.tolerance, initialWeights);
  }

  public Minimizer<DiffFunction> getMinimizer() {
    return getMinimizer(0, null);
  }

  public Minimizer<DiffFunction> getMinimizer(int featurePruneIteration, Evaluator[] evaluators) {
    Minimizer<DiffFunction> minimizer = null;
    QNMinimizer qnMinimizer = null;

    if (flags.useQN || flags.useSGDtoQN) {
      // share code for creation of QNMinimizer
      int qnMem;
      if (featurePruneIteration == 0) {
        qnMem = flags.QNsize;

View on GitHub (pinned to 1b7edd19c4)