{"record":{"id":"90c3c2268064f598","repo":"dgraph-io/dgraph","slug":"duplicate-tokenizers-present-for-attr-s","errorCode":null,"errorMessage":"Duplicate tokenizers present for attr %s","messagePattern":"Duplicate tokenizers present for attr (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"schema/parse.go","lineNumber":455,"sourceCode":"\t\t}\n\t\t// check for valid tokenizer types and duplicates\n\t\tvar seen = make(map[string]bool)\n\t\tvar seenSortableTok bool\n\t\tfor _, t := range schema.Tokenizer {\n\t\t\ttokenizer, has := tok.GetTokenizer(t)\n\t\t\tif !has {\n\t\t\t\treturn errors.Errorf(\"Invalid tokenizer %s\", t)\n\t\t\t}\n\t\t\ttokenizerType, ok := types.TypeForName(tokenizer.Type())\n\t\t\tx.AssertTrue(ok) // Type is validated during tokenizer loading.\n\t\t\tif tokenizerType != typ {\n\t\t\t\treturn errors.Errorf(\"Tokenizer: %s isn't valid for predicate: %s of type: %s\",\n\t\t\t\t\ttokenizer.Name(), x.ParseAttr(schema.Predicate), typ.Name())\n\t\t\t}\n\t\t\tif _, ok := seen[tokenizer.Name()]; !ok {\n\t\t\t\tseen[tokenizer.Name()] = true\n\t\t\t} else {\n\t\t\t\treturn errors.Errorf(\"Duplicate tokenizers present for attr %s\",\n\t\t\t\t\tx.ParseAttr(schema.Predicate))\n\t\t\t}\n\t\t\tif tokenizer.IsSortable() {\n\t\t\t\tif seenSortableTok {\n\t\t\t\t\treturn errors.Errorf(\"More than one sortable index encountered for: %v\",\n\t\t\t\t\t\tschema.Predicate)\n\t\t\t\t}\n\t\t\t\tseenSortableTok = true\n\t\t\t}\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc parseTypeDeclaration(it *lex.ItemIterator, ns uint64) (*pb.TypeUpdate, error) {\n\t// Iterator is currently on the token corresponding to the keyword type.\n\tif it.Item().Typ != itemText || it.Item().Val != \"type\" {\n\t\treturn nil, it.Item().Errorf(\"Expected type keyword. Got %v\", it.Item().Val)","sourceCodeStart":437,"sourceCodeEnd":473,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/schema/parse.go#L437-L473","documentation":"A predicate's @index directive may list each tokenizer only once; resolveTokenizers tracks seen tokenizer names and rejects duplicates with \"Duplicate tokenizers present for attr <pred>\". Duplicate tokenizers would build identical index keys twice.","triggerScenarios":"Schema alteration like `pred: string @index(hash, hash) .` or a SchemaUpdate whose Tokenizer array contains the same name twice.","commonSituations":"Programmatic schema generation appending tokenizers from multiple sources (defaults + user config) without deduplication; merge conflicts in schema files that duplicated a list entry; scripts that concatenate index directives.","solutions":["Deduplicate the tokenizer list in the @index directive.","In programmatic clients, normalize schema.Tokenizer with a set before sending the SchemaUpdate.","Re-apply the alteration."],"exampleFix":"// before\nname: string @index(term, term) .\n// after\nname: string @index(term) .","handlingStrategy":"validation","validationCode":"function dedupeTokenizers(line) {\n  return line.replace(/@index\\s*\\(([^)]*)\\)/, (_, list) => {\n    const uniq = [...new Set(list.split(',').map(s => s.trim()))];\n    return '@index(' + uniq.join(', ') + ')';\n  });\n}","typeGuard":null,"tryCatchPattern":"try {\n  await dgraph.Alter(ctx, op);\n} catch (e) {\n  if (String(e).includes('Duplicate tokenizers')) {\n    // dedupe Tokenizer array / @index list and retry\n  }\n  throw e;\n}","preventionTips":["Deduplicate tokenizer arrays before sending SchemaUpdate","Deduplicate @index lists in generated schema","Watch for merges that duplicate comma-separated entries"],"tags":["dgraph","schema","tokenizer","duplicate"],"backgroundTag":"schema-validation-failed","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}