{"record":{"id":"2b65f45d4ec26f9f","repo":"instructure/canvas-lms","slug":"canvas-http-cb-trip-on-domain","errorCode":null,"errorMessage":"CANVAS_HTTP CB_TRIP ON #{domain}","messagePattern":"CANVAS_HTTP CB_TRIP ON #(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"gems/canvas_http/lib/canvas_http/circuit_breaker.rb","lineNumber":53,"sourceCode":"\n      def tripped?(domain)\n        return false if redis_client.nil?\n\n        !redis_client.get(tripped_key(domain), failsafe: nil).nil?\n      end\n\n      def trip_if_necessary(domain)\n        return if redis_client.nil?\n\n        key = threshold_key(domain)\n        current_count = redis_client.pipelined(key) do |pipeline|\n          pipeline.setnx(key, 0)\n          pipeline.expire(key, WINDOW)\n          pipeline.incr(key)\n        end.last\n        if current_count > THRESHOLD\n          redis_client.setex(tripped_key(domain), INTERVAL, \"1\")\n          CanvasHttp.logger.warn(\"CANVAS_HTTP CB_TRIP ON #{domain}\")\n        end\n      rescue Redis::BaseConnectionError\n        # ignore\n      end\n\n      def tripped_key(domain)\n        \"http_cb_tripped_#{domain}\"\n      end\n\n      def threshold_key(domain)\n        \"http_cb_counter_#{domain}\"\n      end\n\n      def redis_client\n        @redis.respond_to?(:call) ? @redis.call : @redis || nil\n      end\n    end\n  end","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/instructure/canvas-lms/blob/1c9f0bb8013ed69c4f2efe11fd483025469b7e6c/gems/canvas_http/lib/canvas_http/circuit_breaker.rb#L35-L71","documentation":"CanvasHttp's circuit breaker tracks failed requests per domain in Redis via a sliding window (setnx/expire/incr). When current_count exceeds THRESHOLD within WINDOW, it sets a tripped key for INTERVAL seconds and logs this warning. While tripped, requests to that domain are refused without hitting the network. Redis errors are deliberately ignored so the breaker fails open.","triggerScenarios":"trip_if_necessary increments the failure counter for domain and current_count > THRESHOLD — i.e., repeated connection failures/timeouts to the same domain within the window — so the breaker trips and blocks further requests to that domain for INTERVAL.","commonSituations":"An external host (e.g., a plugin/service endpoint) is down and Canvas keeps retrying until the breaker trips; network egress issues making all requests to a domain fail; a very low THRESHOLD configured making the breaker trip on modest bursts of transient failures.","solutions":["Fix the underlying connectivity to the failing domain (check it's up, DNS, egress firewall rules)","Wait INTERVAL for the breaker to reset, or clear the Redis tripped key for the domain to restore immediately","Tune THRESHOLD/WINDOW/INTERVAL constants if the breaker trips too aggressively for legitimate traffic","Confirm Redis is reachable — if it is not, the breaker silently fails open and provides no protection"],"exampleFix":"// before\n# repeated failures trip breaker; requests now refused\nCanvasHttp.get('https://flaky.example.com/x')\n// after\n# clear tripped key after remediation, then retry\nredis.del(CanvasHttp::CircuitBreaker.tripped_key('flaky.example.com'))\nCanvasHttp.get('https://flaky.example.com/x')","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"begin\n  CanvasHttp.get(url)\nrescue CanvasHttp::CircuitTrippedError\n  backoff_then_retry(url, delay: CanvasHttp::CircuitBreaker::INTERVAL)\nend","preventionTips":["Monitor the CB_TRIP warning stream to catch failing upstream domains early","Fix underlying connectivity before clearing tripped keys","Keep Redis healthy — an unreachable Redis silently disables the breaker","Size THRESHOLD/WINDOW to your normal traffic burst pattern"],"tags":["circuit-breaker","redis","ssrf","canvas-http"],"backgroundTag":"circuit-breaker-open","analyzedSha":"1c9f0bb8013ed69c4f2efe11fd483025469b7e6c","analyzedAt":"2026-09-15T20:33:18.891Z","contentChangedAt":"2026-09-15T20:33:18.891Z","schemaVersion":2},"datasetVersion":"2026-09-23T02:17:17.105Z"}