{"record":{"id":"987c652b940b79b7","repo":"stanfordnlp/CoreNLP","slug":"after-w-derivative-index-x-length-987c65","errorCode":null,"errorMessage":"after W derivative, index() != x.length()","messagePattern":"after W derivative, index\\(\\) != x\\.length\\(\\)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/ie/crf/CRFNonLinearSecondOrderLogConditionalObjectiveFunction.java","lineNumber":773,"sourceCode":"        for (int j = 0; j < eU4Edge[i].length; j++) {\n          derivative[index++] = (eU4Edge[i][j] - Uhat4Edge[i][j]);\n          if (VERBOSE) {\n            log.info(\"outputLayerWeights4Edge deriv(\" + i + \",\" + j + \") = \" + eU4Edge[i][j] + \" - \" + Uhat4Edge[i][j] + \" = \" + derivative[index - 1]);\n          }\n        }\n      }\n      for (int i = 0; i < eU.length; i++) {\n        for (int j = 0; j < eU[i].length; j++) {\n          derivative[index++] = (eU[i][j] - Uhat[i][j]);\n          if (VERBOSE) {\n            log.info(\"outputLayerWeights deriv(\" + i + \",\" + j + \") = \" + eU[i][j] + \" - \" + Uhat[i][j] + \" = \" + derivative[index - 1]);\n          }\n        }\n      }\n    }\n\n    if (index != x.length)\n      throw new RuntimeException(\"after W derivative, index(\"+index+\") != x.length(\"+x.length+\")\");\n\n    int regSize = x.length;\n    if (flags.skipOutputRegularization || flags.softmaxOutputLayer) {\n      regSize = beforeOutputWeights;\n    }\n\n    // incorporate priors\n    if (prior == QUADRATIC_PRIOR) {\n      double sigmaSq = sigma * sigma;\n      for (int i = 0; i < regSize; i++) {\n        double k = 1.0;\n        double w = x[i];\n        value += k * w * w / 2.0 / sigmaSq;\n        derivative[i] += k * w / sigmaSq;\n      }\n    } else if (prior == HUBER_PRIOR) {\n      double sigmaSq = sigma * sigma;\n      for (int i = 0; i < regSize; i++) {","sourceCodeStart":755,"sourceCodeEnd":791,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/ie/crf/CRFNonLinearSecondOrderLogConditionalObjectiveFunction.java#L755-L791","documentation":"This is an internal sanity check at the end of the CRF's derivative computation for the W (feature weight) part of the objective. After iterating over all documents/positions and writing gradient values into the flattened parameter array `x`, the code asserts that the write cursor `index` has consumed exactly the whole array. If not, gradient computation and the parameter layout disagree, so results would be silently wrong.","triggerScenarios":"Calling CRF training with a non-linear CRF (CRFNonLinearSecondOrderLogConditionalObjectiveFunction) where the flattening order assumed by the derivative loop does not match the weight layout produced at initialization — e.g. mismatched flags such as softmaxOutputLayer/skipOutputRegularization interacting with feature dimensions, or a feature index that changed between setup and gradient calculation.","commonSituations":"Custom feature factories that produce indices outside the declared feature space; configuring output-layer/bias options inconsistently between the CRFLogConditionalObjectiveFunction and the non-linear variant; running training after modifying the dataset so cached feature dimensions are stale.","solutions":["Verify the feature index/dimension used at CRF initialization matches what the feature factory actually emits (no stale or extra features).","Check flags like skipOutputRegularization and softmaxOutputLayer are set consistently across all code paths that build the objective.","Re-run CRF training from scratch (no cached/transferred weights) so all internal dimensions are recomputed together.","If it persists with a custom feature factory, dump the feature index size and the failing `index` value to find which feature overflows the layout."],"exampleFix":"// before (inconsistent flags)\nflags.softmaxOutputLayer = true;\n// objective built with defaults elsewhere\nprops.setProperty(\"softmaxOutputLayer\", \"false\");\n\n// after\nflags.softmaxOutputLayer = true;\nprops.setProperty(\"softmaxOutputLayer\", \"true\"); // keep flag consistent everywhere","handlingStrategy":"validation","validationCode":"// Java: before training, verify feature index matches factory output\nint expected = featureIndex.size();\nfor (List<String> feats : allDocFeatures) {\n  for (String f : feats) {\n    if (featureIndex.indexOf(f) < 0)\n      throw new IllegalStateException(\"feature not in index: \" + f);\n  }\n}\nif (flags.softmaxOutputLayer && flags.skipOutputRegularization)\n  System.err.println(\"WARN: verify regSize handling covers output-layer weights\");","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep flags (softmaxOutputLayer, skipOutputRegularization, window/order) identical across objective construction and training config.","Never reuse cached weights or feature indices across differently-configured CRF runs.","Test training on a tiny dataset first — dimension mismatches surface quickly there.","Pin the Stanford CoreNLP version; CRF internals changed across releases."],"tags":["java","nlp","crf","internal-state"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}