instructure/canvas-lms · error · ConfigError

kafka_events has brokers configured but required topic(s)…

Error message

kafka_events has brokers configured but required topic(s) missing from DynamicSettings: #{missing.join(", ")}

What it means

Canvas::KafkaEvents::Config#validate! checks every required topic key (Events.topic_keys) resolves to a non-empty topic name via topic_for (sourced from DynamicSettings). If brokers are configured but any required topic name is missing, ConfigError is raised listing the missing topic keys.

Solutions

1. Add the listed topic names under kafka_events topics in DynamicSettings for the environment
2. Create the corresponding topics in the Kafka cluster and match names in config
3. Run the canvas task/console to dump Canvas::KafkaEvents::Config required topic keys and verify each is present
4. After adding new event types in an upgrade, re-sync the config tree from the documented sample

Example fix

# before (DynamicSettings kafka_events)
brokers: [kafka1:9092]
// after
brokers: [kafka1:9092]
topics:
  live_events: canvas-live-events
  migrations: canvas-migrations

missing = Canvas::KafkaEvents::Events.topic_keys.reject { |k| config.topic_for(k).present? }
raise "missing topics: #{missing.join(',')}" if missing.any?

ok = topic_keys.all? { |k| topics.is_a?(Hash) && topics[k.to_s].present? }

begin
  Canvas::KafkaEvents::Config.validate!
rescue Canvas::KafkaEvents::ConfigError => e
  provision_missing_topics(e.message)
  raise
end

When it happens

Trigger: Instantiating/validating Config when kafka brokers are set but DynamicSettings lacks topic names for one or more required event topic keys (e.g. after adding a new event type without provisioning its topic key).

Common situations: New Canvas version requires new topics; consul/DynamicSettings tree not updated; typo in topic key under the kafka_events namespace; environment missing the topics: block.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• kafka_events connection requires both sasl_username and…
• unknown keyword(s): #
• A partner ID is required to use Academic Benchmarks
• A partner key is required to use Academic Benchmarks
• an object with an interface for loading settings must be…

AI-assisted analysis of instructure/canvas-lms@1c9f0bb801 (2026-09-15). Data as JSON: /api/errors/094ef947f85c038d. Report an issue: GitHub.

Appendix: source

Thrown at lib/canvas/kafka_events/config.rb:85

        "socket.keepalive.enable": "true",
        "linger.ms": "50",
        "message.timeout.ms": message_timeout_ms.to_s,
        "queue.buffering.max.messages": queue_buffering_max_messages.to_s,
        "reconnect.backoff.ms": reconnect_backoff_ms.to_s,
        "reconnect.backoff.max.ms": reconnect_backoff_max_ms.to_s,
      }.compact
    end

    def validate!
      if sasl_configured? && !sasl_credentials_complete?
        raise ConfigError,
              "kafka_events connection requires both sasl_username and sasl_password when SASL is in use"
      end

      missing = Events.topic_keys.reject { |key| topic_for(key).present? }
      return if missing.empty?

      raise ConfigError,
            "kafka_events has brokers configured but required topic(s) missing from DynamicSettings: #{missing.join(", ")}"
    end

    private

    def load_operational_config
      parsed = YAML.safe_load(DynamicSettings.find(tree: :private)["kafka_events.yml", failsafe: nil] || "{}")
      parsed.is_a?(Hash) ? parsed : {}
    rescue => e
      Rails.logger.warn("Kafka operational settings load error: #{e.message}")
      {}
    end

    def load_connection_config
      Rails.application.credentials.kafka_events&.with_indifferent_access || {}
    end
  end
end

View on GitHub (pinned to 1c9f0bb801)