{"record":{"id":"360a99ea750cbe5a","repo":"stanfordnlp/CoreNLP","slug":"after-w-derivative-index-beforeoutputweights","errorCode":null,"errorMessage":"after W derivative, index() != beforeOutputWeights()","messagePattern":"after W derivative, index\\(\\) != beforeOutputWeights\\(\\)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/ie/crf/CRFNonLinearLogConditionalObjectiveFunction.java","lineNumber":702,"sourceCode":"        if (VERBOSE) {\n          log.info(\"linearWeights deriv(\" + i + \",\" + j + \") = \" + E[i][j] + \" - \" + Ehat[i][j] + \" = \" + derivative[index - 1]);\n        }\n      }\n    }\n    if (index != edgeParamCount)\n      throw new RuntimeException(\"after edge derivative, index(\"+index+\") != edgeParamCount(\"+edgeParamCount+\")\");\n\n    for (int i = 0; i < eW.length; i++) {\n      for (int j = 0; j < eW[i].length; j++) {\n        derivative[index++] = (eW[i][j] - What[i][j]);\n        if (VERBOSE) {\n          log.info(\"inputLayerWeights deriv(\" + i + \",\" + j + \") = \" + eW[i][j] + \" - \" + What[i][j] + \" = \" + derivative[index - 1]);\n        }\n      }\n    }\n\n    if (index != beforeOutputWeights)\n      throw new RuntimeException(\"after W derivative, index(\"+index+\") != beforeOutputWeights(\"+beforeOutputWeights+\")\");\n\n    if (useOutputLayer) {\n      for (int i = 0; i < eU.length; i++) {\n        for (int j = 0; j < eU[i].length; j++) {\n          if (flags.hardcodeSoftmaxOutputWeights)\n            derivative[index++] = 0;\n          else\n            derivative[index++] = (eU[i][j] - Uhat[i][j]);\n          if (VERBOSE) {\n            log.info(\"outputLayerWeights deriv(\" + i + \",\" + j + \") = \" + eU[i][j] + \" - \" + Uhat[i][j] + \" = \" + derivative[index - 1]);\n          }\n        }\n      }\n    }\n\n    if (index != x.length)\n      throw new RuntimeException(\"after W derivative, index(\"+index+\") != x.length(\"+x.length+\")\");\n","sourceCodeStart":684,"sourceCodeEnd":720,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/ie/crf/CRFNonLinearLogConditionalObjectiveFunction.java#L684-L720","documentation":"After writing the input-layer (hidden W matrix) derivative block in CRFNonLinearLogConditionalObjectiveFunction.calculate(), the write index must equal beforeOutputWeights (the offset at which output-layer parameters start). A mismatch means the eW/What loops wrote a different number of entries than inputLayerSize * numClasses predicts, breaking the parameter-vector layout.","triggerScenarios":"Calling calculate() when inputLayerSize was set inconsistently with the actual eW array dimensions -- e.g. flags.sparseOutputLayer or non-default hidden-layer sizing interacting with modified code, or constructor arguments (numNodeFeatures, numClasses) that disagree with the weight arrays.","commonSituations":"Custom extensions of the non-linear CRF (changing hidden layer size or sparsity flags) during CRFClassifier training with useNonLinearCRF=true; merges of upstream patches that desynchronized dimension constants.","solutions":["Confirm inputLayerSize in the flags matches the eW array's first dimension, and that sparseOutputLayer/tieOutputLayer settings are the intended ones.","Check that the W-derivative loops iterate over the full eW matrix (inputLayerSize x numClasses) exactly once.","Rebuild against an unmodified official CoreNLP version to rule out local patches.","Log inputLayerSize, numClasses, and beforeOutputWeights before training to verify they line up."],"exampleFix":"// before (custom hidden layer resized but flags not updated)\nthis.eW = new double[customHiddenSize][numClasses];\n// after (keep flags and array in sync)\nthis.inputLayerSize = customHiddenSize;\nthis.eW = new double[this.inputLayerSize][numClasses];","handlingStrategy":"validation","validationCode":"// verify W block size matches offset math\nif (inputLayerSize * numClasses != beforeOutputWeights - edgeParamCount)\n  throw new IllegalStateException(\"W block size mismatch\");","typeGuard":null,"tryCatchPattern":"try {\n  minimizer.minimize(fn, tol, x);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"!= beforeOutputWeights\")) {\n    // print inputLayerSize/numClasses and reconfigure flags\n  } else throw e;\n}","preventionTips":["Keep flags-driven sizes (inputLayerSize, sparseOutputLayer) in sync with weight arrays","Prefer configuring layers only through SeqClassifierFlags, not by editing arrays","Log dimension constants once before training"],"tags":["crf","gradient","invariant","indexing"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}