{"record":{"id":"179605fd7da24fef","repo":"stanfordnlp/CoreNLP","slug":"gradient-check-failed","errorCode":null,"errorMessage":"gradient check failed","messagePattern":"gradient check failed","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/ie/crf/CRFClassifier.java","lineNumber":1876,"sourceCode":"      }\n    }\n    log.info(\"numWeights: \" + initialWeights.length);\n\n    if (flags.testObjFunction) {\n      StochasticDiffFunctionTester tester = new StochasticDiffFunctionTester(func);\n      if (tester.testSumOfBatches(initialWeights, 1e-4)) {\n        log.info(\"Successfully tested stochastic objective function.\");\n      } else {\n        throw new IllegalStateException(\"Testing of stochastic objective function failed.\");\n      }\n\n    }\n    //check gradient\n    if (flags.checkGradient) {\n      if (func.gradientCheck()) {\n        log.info(\"gradient check passed\");\n      } else {\n        throw new RuntimeException(\"gradient check failed\");\n      }\n    }\n    return minimizer.minimize(func, flags.tolerance, initialWeights);\n  }\n\n  public Minimizer<DiffFunction> getMinimizer() {\n    return getMinimizer(0, null);\n  }\n\n  public Minimizer<DiffFunction> getMinimizer(int featurePruneIteration, Evaluator[] evaluators) {\n    Minimizer<DiffFunction> minimizer = null;\n    QNMinimizer qnMinimizer = null;\n\n    if (flags.useQN || flags.useSGDtoQN) {\n      // share code for creation of QNMinimizer\n      int qnMem;\n      if (featurePruneIteration == 0) {\n        qnMem = flags.QNsize;","sourceCodeStart":1858,"sourceCodeEnd":1894,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/ie/crf/CRFClassifier.java#L1858-L1894","documentation":"With flags.checkGradient=true, training calls func.gradientCheck() to numerically verify the analytic gradient of the CRF objective. A false result (analytic and finite-difference gradients disagree beyond tolerance) throws this RuntimeException. The mismatch usually indicates a broken derivative, bad features, or NaN/Inf values.","triggerScenarios":"Setting checkGradient=true and gradientCheck() returns false — commonly due to NaN/Inf in features or weights, extreme feature scaling, or a modified objective whose gradient is wrong.","commonSituations":"Debugging custom features or objective modifications; numerical instability with huge feature values; accidentally leaving the check enabled for large production runs.","solutions":["Inspect training data for NaN/Inf or extreme feature values and clean/normalize them.","If you modified the objective or gradient code, fix the derivative implementation.","Re-run the check on a tiny model where the mismatch can be diagnosed numerically.","Remove checkGradient once validation passes — it is a debug flag."],"exampleFix":"// before\nprops.setProperty(\"checkGradient\", \"true\"); // throws on mismatch\n// after (once the gradient is validated)\nprops.remove(\"checkGradient\");","handlingStrategy":"try-catch","validationCode":"boolean hasBadValues = features.stream().anyMatch(v -> Double.isNaN(v) || Double.isInfinite(v));\nif (\"true\".equals(props.getProperty(\"checkGradient\", \"false\")) && hasBadValues) {\n  throw new IllegalArgumentException(\"NaN/Inf features will break the gradient check\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  classifier.train(files);\n} catch (RuntimeException e) {\n  if (\"gradient check failed\".equals(e.getMessage())) {\n    log.warn(\"Analytic vs numeric gradient mismatch; inspect objective/features\");\n    props.remove(\"checkGradient\"); // after investigation\n  } else throw e;\n}","preventionTips":["Run gradient checks only on tiny models where failures are diagnosable.","Validate features for NaN/Inf and extreme magnitudes first.","Fix derivative code for any custom objective before proceeding.","Disable checkGradient for production runs."],"tags":["java","gradient","optimizer","crf"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}