apache/kafka · error · ConfigException

Configuration 'name' values must not be duplicated.

Error message

Configuration 'name' values must not be duplicated.

What it means

Thrown by ValidList.ensureValid after it builds Set.copyOf(values) and finds its size smaller than the list size, i.e. the supplied list contains duplicates. The validator exists specifically to enforce uniqueness (ValidList.anyNonDuplicateValues / ValidList.in), so any repeated element is rejected regardless of whether individual values are themselves legal.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:1074

        @Override
        public void ensureValid(final String name, final Object value) {
            if (value == null) {
                if (isNullAllowed)
                    return;
                else
                    throw new ConfigException("Configuration '" + name + "' values must not be null.");
            }

            @SuppressWarnings("unchecked")
            List<Object> values = (List<Object>) value;
            if (!isEmptyAllowed && values.isEmpty()) {
                String validString = this.validString.validStrings.isEmpty() ? "any non-empty value" : this.validString.toString();
                throw new ConfigException("Configuration '" + name + "' must not be empty. Valid values include: " + validString);
            }

            if (Set.copyOf(values).size() != values.size()) {
                throw new ConfigException("Configuration '" + name + "' values must not be duplicated.");
            }

            validateIndividualValues(name, values);
        }

        private void validateIndividualValues(String name, List<Object> values) {
            boolean hasValidStrings = !validString.validStrings.isEmpty();

            for (Object value : values) {
                if (value instanceof String) {
                    String string = (String) value;
                    if (string.isEmpty()) {
                        throw new ConfigException("Configuration '" + name + "' values must not be empty.");
                    }
                    if (hasValidStrings) {
                        validString.ensureValid(name, value);
                    }
                }

View on GitHub (pinned to c31c9215e1)

Solutions

Inspect the named config's value and remove the duplicate entries so each element appears exactly once.
If the duplicates originate from merging multiple config sources, de-duplicate at build time (new LinkedHashSet<>(list)) before assigning.
Verify shell/env expansion is not appending the same value twice (e.g. VAR=$VAR,extra).

Example fix

// before
props.put("my.list.config", "read,write,read");

// after
props.put("my.list.config", "read,write");

Defensive patterns

Strategy: validation

Validate before calling

// Deduplicate list-typed config values before passing them in:
Map<String, Object> safe = new HashMap<>();
for (Map.Entry<String, Object> e : configs.entrySet()) {
    Object v = e.getValue();
    if (v instanceof List<?>) {
        List<?> dedup = ((List<?>) v).stream().distinct().collect(Collectors.toList());
        safe.put(e.getKey(), dedup);
    } else {
        safe.put(e.getKey(), v);
    }
}

Type guard

null

Try / catch

try {
    new KafkaAdmin(props);
} catch (ConfigException ce) {
    if (ce.getMessage().contains("duplicated")) {
        log.warn("Duplicate entries in '{}'; de-duplicating and retrying", ce.getName());
        dedupeAndRetry(props);
    } else throw ce;
}

Prevention

Build list-valued configs from a Set rather than a List when order is irrelevant.
When joining values from multiple sources (file + env), dedupe the final list, not each source.
Write a config linter that calls Set.copyOf(list).size()==list.size() for every List config key.

When it happens

Trigger: A ConfigKey validated with ValidList receives a list containing the same element twice, e.g. "a,b,a" split into [a, b, a]. Triggered during ConfigDef.parse() or any direct call to ValidList.ensureValid(name, values).

Common situations: Comma-separated config values where a value is listed twice by accident (copy-paste, merge of two config files, duplicate env-var injection). Also seen when a list is built programmatically by concatenating sources without de-duplication.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/3a2d36ffb97136b7.json. Report an issue: GitHub.