{"record":{"id":"4c3160274ae203b5","repo":"instructure/canvas-lms","slug":"consul-is-unavailable-because-the-circuit-breaker-has","errorCode":null,"errorMessage":"Consul is unavailable because the circuit breaker has tripped","messagePattern":"Consul is unavailable because the circuit breaker has tripped","errorType":"exception","errorClass":"Diplomat::UnknownStatus","httpStatus":null,"severity":"error","filePath":"gems/dynamic_settings/lib/dynamic_settings/prefix_proxy.rb","lineNumber":159,"sourceCode":"        full_key(key),\n        [tree, service, environment, prefix, key].compact.join(\"/\"),\n      ].uniq\n\n      fallback_keys = [\n        [tree, service, prefix, key].compact.join(\"/\"),\n        full_key(key, global: true),\n        [\"global\", tree, service, prefix, key].compact.join(\"/\"),\n      ].uniq - keys\n\n      # try to get the local cache first right away\n      keys.each do |full_key|\n        result = cache.fetch(CACHE_KEY_PREFIX + full_key)\n        return result if result\n      end\n\n      begin\n        if circuit_breaker&.tripped?\n          raise Diplomat::UnknownStatus, \"Consul is unavailable because the circuit breaker has tripped\"\n        end\n\n        # okay now pre-cache an entire tree\n        tree_key = [tree, service, environment].compact.join(\"/\")\n        # This longer TTL is important for race condition for now.\n        # if the tree JUST expired, we don't want to find\n        # a valid tree, and then no valid subkeys, that makes\n        # nils start popping up in the cache.  Subkeys should\n        # last much longer than it takes to notice the tree key is\n        # expired and trying to replace it.  When the tree writes\n        # are fully atomic, this is much less of a concern,\n        # we could have one ttl again\n        subtree_ttl = ttl * 2\n        cache.fetch(CACHE_KEY_PREFIX + tree_key + \"/\", expires_in: ttl) do\n          values = kv_fetch(tree_key, recurse: true, stale: true)\n          if values.nil?\n            # no sense trying to populate the subkeys\n            # when there's no tree","sourceCodeStart":141,"sourceCodeEnd":177,"githubUrl":"https://github.com/instructure/canvas-lms/blob/1c9f0bb8013ed69c4f2efe11fd483025469b7e6c/gems/dynamic_settings/lib/dynamic_settings/prefix_proxy.rb#L141-L177","documentation":"Before talking to Consul, fetch_without_request_cache consults a circuit breaker; if the breaker has tripped after repeated Consul failures it raises Diplomat::UnknownStatus immediately instead of making a network call. This fails fast so requests don't hang waiting on an unhealthy Consul, and lets the failsafe/on-disk cache paths take over.","triggerScenarios":"Any DynamicSettings fetch while the circuit breaker is tripped, i.e. after the configured number of consecutive Consul request failures/timeouts within the breaker window.","commonSituations":"Consul cluster down or unreachable from a pod/node; network partition or DNS failure; Consul reload causing mass timeouts that trip the breaker; new app boot reaching for Consul before it is up.","solutions":["Restore Consul connectivity/health; the breaker resets once calls succeed again","Restart or wait for the circuit breaker window to reset","Use fetch with `failsafe_cache: true` and a pre-populated failsafe file to boot without Consul","Verify CONSUL_URL / network policy so the app can reach the Consul agent"],"exampleFix":"// before\nval = Canvas::DynamicSettings.find(tree: :canvas)['feature_flag']\n// after\nbegin\n  val = Canvas::DynamicSettings.find(tree: :canvas)['feature_flag']\nrescue Diplomat::UnknownStatus\n  val = nil # fall back to default\nend","handlingStrategy":"fallback","validationCode":"if Canvas::DynamicSettings.circuit_breaker&.tripped?\n  # use defaults/failsafe instead of Consul\nend","typeGuard":null,"tryCatchPattern":"begin\n  val = proxy.fetch(key, failsafe_cache: true)\nrescue Diplomat::UnknownStatus\n  val = DEFAULTS[key]\nend","preventionTips":["Ship a failsafe cache file for boot-time settings","Monitor Consul health and alert before breaker trips","Cache settings per-request to cut Consul calls"],"tags":["consul","circuit-breaker","network","configuration"],"backgroundTag":"circuit-breaker-open","analyzedSha":"1c9f0bb8013ed69c4f2efe11fd483025469b7e6c","analyzedAt":"2026-09-15T20:33:18.891Z","contentChangedAt":"2026-09-15T20:33:18.891Z","schemaVersion":2},"datasetVersion":"2026-09-23T02:17:17.105Z"}