stanfordnlp/CoreNLP · error · RuntimeException

after W derivative, index() != x.length()

Error message

after W derivative, index() != x.length()

What it means

This is an internal sanity check at the end of the CRF's derivative computation for the W (feature weight) part of the objective. After iterating over all documents/positions and writing gradient values into the flattened parameter array `x`, the code asserts that the write cursor `index` has consumed exactly the whole array. If not, gradient computation and the parameter layout disagree, so results would be silently wrong.

Solutions

1. Verify the feature index/dimension used at CRF initialization matches what the feature factory actually emits (no stale or extra features).
2. Check flags like skipOutputRegularization and softmaxOutputLayer are set consistently across all code paths that build the objective.
3. Re-run CRF training from scratch (no cached/transferred weights) so all internal dimensions are recomputed together.
4. If it persists with a custom feature factory, dump the feature index size and the failing `index` value to find which feature overflows the layout.

Example fix

// before (inconsistent flags)
flags.softmaxOutputLayer = true;
// objective built with defaults elsewhere
props.setProperty("softmaxOutputLayer", "false");

// after
flags.softmaxOutputLayer = true;
props.setProperty("softmaxOutputLayer", "true"); // keep flag consistent everywhere

// Java: before training, verify feature index matches factory output
int expected = featureIndex.size();
for (List<String> feats : allDocFeatures) {
  for (String f : feats) {
    if (featureIndex.indexOf(f) < 0)
      throw new IllegalStateException("feature not in index: " + f);
  }
}
if (flags.softmaxOutputLayer && flags.skipOutputRegularization)
  System.err.println("WARN: verify regSize handling covers output-layer weights");

When it happens

Trigger: Calling CRF training with a non-linear CRF (CRFNonLinearSecondOrderLogConditionalObjectiveFunction) where the flattening order assumed by the derivative loop does not match the weight layout produced at initialization — e.g. mismatched flags such as softmaxOutputLayer/skipOutputRegularization interacting with feature dimensions, or a feature index that changed between setup and gradient calculation.

Common situations: Custom feature factories that produce indices outside the declared feature space; configuring output-layer/bias options inconsistently between the CRFLogConditionalObjectiveFunction and the non-linear variant; running training after modifying the dataset so cached feature dimensions are stale.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Unknown inference type: " + flags.inferenceType + ". Your op
• conditionalLogProbGivenPrevious requires given one less than
• conditionalLogProbsGivenPrevious requires given one less tha
• conditionalLogProbGivenFirst requires of one less than cliqu
• unnormalizedConditionalLogProbGivenFirst requires of one les

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/987c652b940b79b7. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/ie/crf/CRFNonLinearSecondOrderLogConditionalObjectiveFunction.java:773

        for (int j = 0; j < eU4Edge[i].length; j++) {
          derivative[index++] = (eU4Edge[i][j] - Uhat4Edge[i][j]);
          if (VERBOSE) {
            log.info("outputLayerWeights4Edge deriv(" + i + "," + j + ") = " + eU4Edge[i][j] + " - " + Uhat4Edge[i][j] + " = " + derivative[index - 1]);
          }
        }
      }
      for (int i = 0; i < eU.length; i++) {
        for (int j = 0; j < eU[i].length; j++) {
          derivative[index++] = (eU[i][j] - Uhat[i][j]);
          if (VERBOSE) {
            log.info("outputLayerWeights deriv(" + i + "," + j + ") = " + eU[i][j] + " - " + Uhat[i][j] + " = " + derivative[index - 1]);
          }
        }
      }
    }

    if (index != x.length)
      throw new RuntimeException("after W derivative, index("+index+") != x.length("+x.length+")");

    int regSize = x.length;
    if (flags.skipOutputRegularization || flags.softmaxOutputLayer) {
      regSize = beforeOutputWeights;
    }

    // incorporate priors
    if (prior == QUADRATIC_PRIOR) {
      double sigmaSq = sigma * sigma;
      for (int i = 0; i < regSize; i++) {
        double k = 1.0;
        double w = x[i];
        value += k * w * w / 2.0 / sigmaSq;
        derivative[i] += k * w / sigmaSq;
      }
    } else if (prior == HUBER_PRIOR) {
      double sigmaSq = sigma * sigma;
      for (int i = 0; i < regSize; i++) {

View on GitHub (pinned to 1b7edd19c4)