{"record":{"id":"bcef1e5113bada4a","repo":"stanfordnlp/CoreNLP","slug":"after-w-derivative-index-x-length","errorCode":null,"errorMessage":"after W derivative, index() != x.length()","messagePattern":"after W derivative, index\\(\\) != x\\.length\\(\\)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/ie/crf/CRFNonLinearLogConditionalObjectiveFunction.java","lineNumber":719,"sourceCode":"    if (index != beforeOutputWeights)\n      throw new RuntimeException(\"after W derivative, index(\"+index+\") != beforeOutputWeights(\"+beforeOutputWeights+\")\");\n\n    if (useOutputLayer) {\n      for (int i = 0; i < eU.length; i++) {\n        for (int j = 0; j < eU[i].length; j++) {\n          if (flags.hardcodeSoftmaxOutputWeights)\n            derivative[index++] = 0;\n          else\n            derivative[index++] = (eU[i][j] - Uhat[i][j]);\n          if (VERBOSE) {\n            log.info(\"outputLayerWeights deriv(\" + i + \",\" + j + \") = \" + eU[i][j] + \" - \" + Uhat[i][j] + \" = \" + derivative[index - 1]);\n          }\n        }\n      }\n    }\n\n    if (index != x.length)\n      throw new RuntimeException(\"after W derivative, index(\"+index+\") != x.length(\"+x.length+\")\");\n\n    int regSize = x.length;\n    if (flags.skipOutputRegularization || flags.softmaxOutputLayer || flags.hardcodeSoftmaxOutputWeights) {\n      regSize = beforeOutputWeights;\n    }\n\n    if (DEBUG) log.info(\"done!\");\n\n    if (DEBUG) log.info(\"incorporating priors ...\");\n\n    // incorporate priors\n    if (prior == QUADRATIC_PRIOR) {\n      double sigmaSq = sigma * sigma;\n      double twoSigmaSq =  2.0 * sigmaSq;\n      double w = 0;\n      double valueSum = 0;\n      for (int i = 0; i < regSize; i++) {\n        w = x[i];","sourceCodeStart":701,"sourceCodeEnd":737,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/ie/crf/CRFNonLinearLogConditionalObjectiveFunction.java#L701-L737","documentation":"Final invariant check in CRFNonLinearLogConditionalObjectiveFunction.calculate(): after all derivative blocks (edges, input layer W, and optional output layer U) are written, the write index must equal x.length (the total domain dimension). Failing means fewer or more gradient entries were produced than there are parameters, i.e. derivative and parameter vectors are misaligned.","triggerScenarios":"calculate() called with an x/derivative whose length does not match the internally computed domainDimension() -- e.g. output-layer U loops skipped or truncated (hardcodeSoftmaxOutputWeights / skipOutputRegularization / softmaxOutputLayer flag combinations) while dimension accounting still expects the full matrix.","commonSituations":"Flag combinations around the output layer (softmaxOutputLayer, hardcodeSoftmaxOutputWeights, sparseOutputLayer) that were not fully supported in the code path being used; patched or extended CRF training code during NER model tuning.","solutions":["Avoid untested combinations of softmaxOutputLayer / hardcodeSoftmaxOutputWeights / skipOutputRegularization; enable only one and retest.","Ensure every derivative block (E, eW, eU) is written, including the zeroed entries required when hardcodeSoftmaxOutputWeights is set.","Check that the derivative array passed in was allocated with domainDimension() elements by the optimizer.","Use an unmodified CoreNLP build; this is a layout invariant that only breaks with altered code or exotic flags."],"exampleFix":"// before (skipping output layer zeros when flag set)\nif (flags.hardcodeSoftmaxOutputWeights) { /* nothing written */ }\n// after (must still write zeros to keep index aligned)\nif (flags.hardcodeSoftmaxOutputWeights) {\n  for (int i = 0; i < eU.length; i++)\n    for (int j = 0; j < eU[i].length; j++) derivative[index++] = 0;\n}","handlingStrategy":"validation","validationCode":"// before handing derivative to optimizer\nif (derivative.length != x.length)\n  throw new IllegalStateException(\"gradient/parameter length mismatch: \" + derivative.length + \" vs \" + x.length);","typeGuard":null,"tryCatchPattern":"try {\n  minimizer.minimize(fn, tol, x);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"index(\" ) && e.getMessage().contains(\"x.length\")) {\n    // audit output-layer flags and retrain with default configuration\n  } else throw e;\n}","preventionTips":["Test only one non-default output-layer flag at a time","Ensure all derivative blocks are written, including deliberate zeros","Run a numerical-gradient check on a tiny dataset before full training"],"tags":["crf","gradient","invariant","indexing"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}