apache/druid · error · org.apache.druid.java.util.common.IAE

Cannot set kafka property [auto.offset.reset]. Property…

Error message

Cannot set kafka property [auto.offset.reset]. Property will be forced to [earliest]. Found [%s]

What it means

verifyKafkaProperties() rejects a user-supplied auto.offset.reset because the factory forces 'earliest' so lookup data is fully loaded from the beginning of the topic; any other value would risk an incomplete lookup.

Solutions

Remove auto.offset.reset from kafkaProperties
Rely on the forced 'earliest' behavior to consume the topic from the beginning

Example fix

// before
"auto.offset.reset": "latest"
// after
(removed; Druid forces earliest)

Defensive patterns

Strategy: validation

Validate before calling

if (kafkaProperties.containsKey("auto.offset.reset")) {
  kafkaProperties.remove("auto.offset.reset"); // lookups always start at earliest
}

Prevention

Remember lookups force earliest offsets
Sanitize kafkaProperties before constructing the factory
Use a shared config builder that whitelists allowed consumer keys

When it happens

Trigger: Calling start() when kafkaProperties contains 'auto.offset.reset' with any value (e.g. 'latest').

Common situations: Reusing an ingestion-style Kafka consumer config in a lookup spec where offset reset semantics differ.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/9e6a84a348f3c890. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/kafka-extraction-namespace/src/main/java/org/apache/druid/query/lookup/KafkaLookupExtractorFactory.java:400

  }

  /**
   * Check that the user has not set forbidden Kafka consumer props
   *
   * Some consumer properties must be set in order to guarantee that
   * the consumer will consume the entire topic from the beginning.
   * Otherwise, lookup data may not be loaded completely.
   */
  private void verifyKafkaProperties()
  {
    if (kafkaProperties.containsKey(ConsumerConfig.GROUP_ID_CONFIG)) {
      throw new IAE(
              "Cannot set kafka property [group.id]. Property is randomly generated for you. Found [%s]",
              kafkaProperties.get(ConsumerConfig.GROUP_ID_CONFIG)
      );
    }
    if (kafkaProperties.containsKey(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG)) {
      throw new IAE(
              "Cannot set kafka property [auto.offset.reset]. Property will be forced to [earliest]. Found [%s]",
              kafkaProperties.get(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG)
      );
    }
    if (kafkaProperties.containsKey(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG) &&
          !kafkaProperties.get(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG).equals("false")) {
      throw new IAE(
          "Cannot set kafka property [enable.auto.commit]. Property will be forced to [false]. Found [%s]",
          kafkaProperties.get(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG)
      );
    }
    Preconditions.checkNotNull(
            kafkaProperties.get(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG),
            "bootstrap.servers required property"
    );
  }

  // Overridden in tests

View on GitHub (pinned to 9b90983fd2)