What causes "must be non-null." in apache/kafka?

apache/kafka · error · ConfigException

must be non-null.

Error message

must be non-null.

What it means

Thrown by ConsumerConfig.appendDeserializerToConfig when both the keyDeserializer argument is null AND no value is set for key.deserializer.class in the config map. The consumer requires a way to deserialize keys; it must be supplied either as an instance or a class name. The thrown type is ConfigException.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java:798

            String groupInstanceId = this.getString(GROUP_INSTANCE_ID_CONFIG);
            if (groupInstanceId != null)
                JoinGroupRequest.validateGroupInstanceId(groupInstanceId);

            String groupInstanceIdPart = groupInstanceId != null ? groupInstanceId : CONSUMER_CLIENT_ID_SEQUENCE.getAndIncrement() + "";
            String generatedClientId = String.format("consumer-%s-%s", groupId, groupInstanceIdPart);
            configs.put(CLIENT_ID_CONFIG, generatedClientId);
        }
    }

    public static Map<String, Object> appendDeserializerToConfig(Map<String, Object> configs,
                                                                 Deserializer<?> keyDeserializer,
                                                                 Deserializer<?> valueDeserializer) {
        // validate deserializer configuration, if the passed deserializer instance is null, the user must explicitly set a valid deserializer configuration value
        Map<String, Object> newConfigs = new HashMap<>(configs);
        if (keyDeserializer != null)
            newConfigs.put(KEY_DESERIALIZER_CLASS_CONFIG, keyDeserializer.getClass());
        else if (newConfigs.get(KEY_DESERIALIZER_CLASS_CONFIG) == null)
            throw new ConfigException(KEY_DESERIALIZER_CLASS_CONFIG, null, "must be non-null.");
        if (valueDeserializer != null)
            newConfigs.put(VALUE_DESERIALIZER_CLASS_CONFIG, valueDeserializer.getClass());
        else if (newConfigs.get(VALUE_DESERIALIZER_CLASS_CONFIG) == null)
            throw new ConfigException(VALUE_DESERIALIZER_CLASS_CONFIG, null, "must be non-null.");
        return newConfigs;
    }

    private void maybeOverrideEnableAutoCommit(Map<String, Object> configs) {
        Optional<String> groupId = Optional.ofNullable(getString(CommonClientConfigs.GROUP_ID_CONFIG));
        Map<String, Object> originals = originals();
        boolean enableAutoCommit = originals.containsKey(ENABLE_AUTO_COMMIT_CONFIG) ? getBoolean(ENABLE_AUTO_COMMIT_CONFIG) : false;
        if (groupId.isEmpty()) { // overwrite in case of default group id where the config is not explicitly provided
            if (!originals.containsKey(ENABLE_AUTO_COMMIT_CONFIG)) {
                configs.put(ENABLE_AUTO_COMMIT_CONFIG, false);
            } else if (enableAutoCommit) {
                throw new InvalidConfigurationException(ENABLE_AUTO_COMMIT_CONFIG + " cannot be set to true when default group id (null) is used.");
            }
        }

View on GitHub (pinned to 996fb4585a)

Solutions

Set key.deserializer in your Properties, e.g. props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()).
Or pass a Deserializer instance to the KafkaConsumer constructor for keys.
Use a helper that guarantees both deserializers are configured before constructing the consumer.

Example fix

// before
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("value.deserializer", StringDeserializer.class.getName());
new KafkaConsumer<>(props);

// after
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
new KafkaConsumer<>(props);

Defensive patterns

Strategy: validation

Validate before calling

if (keyDeserializer == null && !configs.containsKey(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG)) {
    throw new ConfigException("key.deserializer must be set or a key Deserializer instance provided");
}

Type guard

static boolean hasKeyDeserializer(Map<String,Object> configs, Deserializer<?> d) {
    return d != null || configs.containsKey(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG);
}

Try / catch

try {
    consumer = new KafkaConsumer<>(props);
} catch (ConfigException e) {
    if (e.getMessage().contains("key.deserializer")) {
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        consumer = new KafkaConsumer<>(props);
    } else throw e;
}

Prevention

Always set both key.deserializer and value.deserializer in consumer Properties.
Centralize consumer construction so deserializers cannot be forgotten.
Add a config sanity test to your test suite.

When it happens

Trigger: Calling KafkaConsumer(props, null, valueDeserializer) while props also lacks "key.deserializer"; constructing a KafkaConsumer with Properties that omit key.deserializer and passing no key deserializer instance.

Common situations: First-time consumer setup forgetting the key.deserializer property; copy-pasting a consumer config template that only had value.deserializer; setting the property to an empty string (treated as not set).

Related errors

AI-assisted analysis of apache/kafka@996fb4585a (2026-08-11). Data as JSON: /api/errors/e2b69adf033822fe. Report an issue: GitHub.