instructure/canvas-lms · error · Diplomat::UnknownStatus

Consul is unavailable because the circuit breaker has…

Error message

Consul is unavailable because the circuit breaker has tripped

What it means

Before talking to Consul, fetch_without_request_cache consults a circuit breaker; if the breaker has tripped after repeated Consul failures it raises Diplomat::UnknownStatus immediately instead of making a network call. This fails fast so requests don't hang waiting on an unhealthy Consul, and lets the failsafe/on-disk cache paths take over.

Solutions

1. Restore Consul connectivity/health; the breaker resets once calls succeed again
2. Restart or wait for the circuit breaker window to reset
3. Use fetch with `failsafe_cache: true` and a pre-populated failsafe file to boot without Consul
4. Verify CONSUL_URL / network policy so the app can reach the Consul agent

Example fix

// before
val = Canvas::DynamicSettings.find(tree: :canvas)['feature_flag']
// after
begin
  val = Canvas::DynamicSettings.find(tree: :canvas)['feature_flag']
rescue Diplomat::UnknownStatus
  val = nil # fall back to default
end

if Canvas::DynamicSettings.circuit_breaker&.tripped?
  # use defaults/failsafe instead of Consul
end

begin
  val = proxy.fetch(key, failsafe_cache: true)
rescue Diplomat::UnknownStatus
  val = DEFAULTS[key]
end

When it happens

Trigger: Any DynamicSettings fetch while the circuit breaker is tripped, i.e. after the configured number of consecutive Consul request failures/timeouts within the breaker window.

Common situations: Consul cluster down or unreachable from a pod/node; network partition or DNS failure; Consul reload causing mass timeouts that trip the breaker; new app boot reaching for Consul before it is up.

Related errors

• A partner ID is required to use Academic Benchmarks
• A partner key is required to use Academic Benchmarks
• an object with an interface for loading settings must be…
• an object with an interface for loading settings must be…
• CANVAS_HTTP CB_TRIP ON #

AI-assisted analysis of instructure/canvas-lms@1c9f0bb801 (2026-09-15). Data as JSON: /api/errors/4c3160274ae203b5. Report an issue: GitHub.

Appendix: source

Thrown at gems/dynamic_settings/lib/dynamic_settings/prefix_proxy.rb:159

        full_key(key),
        [tree, service, environment, prefix, key].compact.join("/"),
      ].uniq

      fallback_keys = [
        [tree, service, prefix, key].compact.join("/"),
        full_key(key, global: true),
        ["global", tree, service, prefix, key].compact.join("/"),
      ].uniq - keys

      # try to get the local cache first right away
      keys.each do |full_key|
        result = cache.fetch(CACHE_KEY_PREFIX + full_key)
        return result if result
      end

      begin
        if circuit_breaker&.tripped?
          raise Diplomat::UnknownStatus, "Consul is unavailable because the circuit breaker has tripped"
        end

        # okay now pre-cache an entire tree
        tree_key = [tree, service, environment].compact.join("/")
        # This longer TTL is important for race condition for now.
        # if the tree JUST expired, we don't want to find
        # a valid tree, and then no valid subkeys, that makes
        # nils start popping up in the cache.  Subkeys should
        # last much longer than it takes to notice the tree key is
        # expired and trying to replace it.  When the tree writes
        # are fully atomic, this is much less of a concern,
        # we could have one ttl again
        subtree_ttl = ttl * 2
        cache.fetch(CACHE_KEY_PREFIX + tree_key + "/", expires_in: ttl) do
          values = kv_fetch(tree_key, recurse: true, stale: true)
          if values.nil?
            # no sense trying to populate the subkeys
            # when there's no tree

View on GitHub (pinned to 1c9f0bb801)