dgraph-io/dgraph · error

Duplicate tokenizers present for attr %s

Error message

Duplicate tokenizers present for attr %s

What it means

A predicate's @index directive may list each tokenizer only once; resolveTokenizers tracks seen tokenizer names and rejects duplicates with "Duplicate tokenizers present for attr <pred>". Duplicate tokenizers would build identical index keys twice.

Source

Thrown at schema/parse.go:455

		}
		// check for valid tokenizer types and duplicates
		var seen = make(map[string]bool)
		var seenSortableTok bool
		for _, t := range schema.Tokenizer {
			tokenizer, has := tok.GetTokenizer(t)
			if !has {
				return errors.Errorf("Invalid tokenizer %s", t)
			}
			tokenizerType, ok := types.TypeForName(tokenizer.Type())
			x.AssertTrue(ok) // Type is validated during tokenizer loading.
			if tokenizerType != typ {
				return errors.Errorf("Tokenizer: %s isn't valid for predicate: %s of type: %s",
					tokenizer.Name(), x.ParseAttr(schema.Predicate), typ.Name())
			}
			if _, ok := seen[tokenizer.Name()]; !ok {
				seen[tokenizer.Name()] = true
			} else {
				return errors.Errorf("Duplicate tokenizers present for attr %s",
					x.ParseAttr(schema.Predicate))
			}
			if tokenizer.IsSortable() {
				if seenSortableTok {
					return errors.Errorf("More than one sortable index encountered for: %v",
						schema.Predicate)
				}
				seenSortableTok = true
			}
		}
	}
	return nil
}

func parseTypeDeclaration(it *lex.ItemIterator, ns uint64) (*pb.TypeUpdate, error) {
	// Iterator is currently on the token corresponding to the keyword type.
	if it.Item().Typ != itemText || it.Item().Val != "type" {
		return nil, it.Item().Errorf("Expected type keyword. Got %v", it.Item().Val)

View on GitHub (pinned to 759e242be6)

Solutions

1. Deduplicate the tokenizer list in the @index directive.
2. In programmatic clients, normalize schema.Tokenizer with a set before sending the SchemaUpdate.
3. Re-apply the alteration.

Example fix

// before
name: string @index(term, term) .
// after
name: string @index(term) .

function dedupeTokenizers(line) {
  return line.replace(/@index\s*\(([^)]*)\)/, (_, list) => {
    const uniq = [...new Set(list.split(',').map(s => s.trim()))];
    return '@index(' + uniq.join(', ') + ')';
  });
}

try {
  await dgraph.Alter(ctx, op);
} catch (e) {
  if (String(e).includes('Duplicate tokenizers')) {
    // dedupe Tokenizer array / @index list and retry
  }
  throw e;
}

When it happens

Trigger: Schema alteration like `pred: string @index(hash, hash) .` or a SchemaUpdate whose Tokenizer array contains the same name twice.

Common situations: Programmatic schema generation appending tokenizers from multiple sources (defaults + user config) without deduplication; merge conflicts in schema files that duplicated a list entry; scripts that concatenate index directives.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• illegal rune found "%c", expecting {
• Malformed JSON
• Require type of tokenizer for pred: %s of type: %s for index
• Tokenizers present without indexing on attr %s
• Invalid tokenizer %s

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/90c3c2268064f598. Report an issue: GitHub.