{"record":{"id":"789abc8b23e182fd","repo":"stanfordnlp/CoreNLP","slug":"annotatorname-replacing-duplicate-entry-highe","errorCode":null,"errorMessage":"${annotatorName}: Replacing duplicate entry (higher priority): old=${oldEntry}, new=${entry}","messagePattern":"(.+?): Replacing duplicate entry \\(higher priority\\): old=(.+?), new=(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/edu/stanford/nlp/pipeline/TokensRegexNERAnnotator.java","lineNumber":825,"sourceCode":"      for (int i = 0; i < types.length; i++) {\n        String type = types[i];\n        // TODO: Have option to allow commas in types\n        int commaPos = type.indexOf(',');\n        if (commaPos > 0) {\n          // Strip the \",\" and just take first type\n          String newType = type.substring(0, commaPos).trim();\n          logger.warn(annotatorName + \": Entry has multiple types for \" +\n                  annotationFieldnames[i] + \": \" + line + \".  Taking type to be \" + newType);\n          types[i] = newType;\n        }\n      }\n\n      Entry entry = new Entry(tokensRegex, regexes, types, overwritableTypes, priority, weight, annotateGroup);\n\n      if (seenRegexes.containsKey(Arrays.asList(key))) {\n        Entry oldEntry = seenRegexes.get(key);\n        if (priority > oldEntry.priority) {\n          logger.warn(annotatorName +\n                  \": Replacing duplicate entry (higher priority): old=\" + oldEntry + \", new=\" + entry);\n        } else {\n          String oldTypeDesc = oldEntry.getTypeDescription();\n          String newTypeDesc = entry.getTypeDescription();\n          if (!oldTypeDesc.equals(newTypeDesc)) {\n            if (verbose) {\n              logger.warn(annotatorName + \": Ignoring duplicate entry: \" +\n                      split[0] + \", old type = \" + oldTypeDesc + \", new type = \" + newTypeDesc);\n            }\n          // } else {\n          //   if (verbose) {\n          //     logger.warn(annotatorName + \": Duplicate entry [ignored]: \" +\n          //             split[0] + \", old type = \" + oldEntry.type + \", new type = \" + type);\n          //   }\n          }\n          continue;\n        }\n      }","sourceCodeStart":807,"sourceCodeEnd":843,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/TokensRegexNERAnnotator.java#L807-L843","documentation":"When a TokensRegexNER rule file contains two entries with the same regex key, the annotator keeps one: if the new entry has a strictly higher priority, it replaces the old one and this warning is logged showing both entries. This prevents silent duplication but signals your rule file has duplicate keys.","triggerScenarios":"The same tokens regex (key) appears twice in a mapping file with different priorities; the later, higher-priority entry overwrites the earlier one at load time.","commonSituations":"Appending new rules to an existing file without removing old duplicates; merging rule files from different teams; copy-paste duplication in hand-maintained gazetteers.","solutions":["Search the mapping file for duplicate keys and delete the superseded entry","If both rules are intended, differentiate the regex keys or adjust priorities deliberately","Keep rules files deduplicated programmatically (e.g. sort/unique on the first column) as a build step"],"exampleFix":"// before (mapping file)\nBarack\tPERSON\t0.0\nBarack\tPERSON\t1.0\n// after\nBarack\tPERSON\t1.0","handlingStrategy":"validation","validationCode":"Set<String> seen = new HashSet<>();\nfor (String line : Files.readAllLines(rulesPath)) {\n  String key = line.split(\"\\t\")[0];\n  if (!seen.add(key))\n    throw new IllegalArgumentException(\"Duplicate rule key: \" + key);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Deduplicate rule keys before shipping rules files","Merge rule sources deliberately, not by concatenation","Keep a canonical rules file under version control","Watch startup warnings for duplicate-entry messages in CI"],"tags":["nlp","ner","duplicate-keys","configuration"],"backgroundTag":"duplicate-entry","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}