instructure/canvas-lms · warning

CANVAS_HTTP CB_TRIP ON #

Error message

CANVAS_HTTP CB_TRIP ON #{domain}

What it means

CanvasHttp's circuit breaker tracks failed requests per domain in Redis via a sliding window (setnx/expire/incr). When current_count exceeds THRESHOLD within WINDOW, it sets a tripped key for INTERVAL seconds and logs this warning. While tripped, requests to that domain are refused without hitting the network. Redis errors are deliberately ignored so the breaker fails open.

Solutions

1. Fix the underlying connectivity to the failing domain (check it's up, DNS, egress firewall rules)
2. Wait INTERVAL for the breaker to reset, or clear the Redis tripped key for the domain to restore immediately
3. Tune THRESHOLD/WINDOW/INTERVAL constants if the breaker trips too aggressively for legitimate traffic
4. Confirm Redis is reachable — if it is not, the breaker silently fails open and provides no protection

Example fix

// before
# repeated failures trip breaker; requests now refused
CanvasHttp.get('https://flaky.example.com/x')
// after
# clear tripped key after remediation, then retry
redis.del(CanvasHttp::CircuitBreaker.tripped_key('flaky.example.com'))
CanvasHttp.get('https://flaky.example.com/x')

begin
  CanvasHttp.get(url)
rescue CanvasHttp::CircuitTrippedError
  backoff_then_retry(url, delay: CanvasHttp::CircuitBreaker::INTERVAL)
end

When it happens

Trigger: trip_if_necessary increments the failure counter for domain and current_count > THRESHOLD — i.e., repeated connection failures/timeouts to the same domain within the window — so the breaker trips and blocks further requests to that domain for INTERVAL.

Common situations: An external host (e.g., a plugin/service endpoint) is down and Canvas keeps retrying until the breaker trips; network egress issues making all requests to a domain fail; a very low THRESHOLD configured making the breaker trip on modest bursts of transient failures.

Related errors

• CANVAS_HTTP WARNING | host: #
• CANVAS_HTTP WARNING insecure address | host: #
• an object with an interface for loading settings must be…
• Consul is unavailable because the circuit breaker has…
• max number of clients reached

AI-assisted analysis of instructure/canvas-lms@1c9f0bb801 (2026-09-15). Data as JSON: /api/errors/2b65f45d4ec26f9f. Report an issue: GitHub.

Appendix: source

Thrown at gems/canvas_http/lib/canvas_http/circuit_breaker.rb:53

      def tripped?(domain)
        return false if redis_client.nil?

        !redis_client.get(tripped_key(domain), failsafe: nil).nil?
      end

      def trip_if_necessary(domain)
        return if redis_client.nil?

        key = threshold_key(domain)
        current_count = redis_client.pipelined(key) do |pipeline|
          pipeline.setnx(key, 0)
          pipeline.expire(key, WINDOW)
          pipeline.incr(key)
        end.last
        if current_count > THRESHOLD
          redis_client.setex(tripped_key(domain), INTERVAL, "1")
          CanvasHttp.logger.warn("CANVAS_HTTP CB_TRIP ON #{domain}")
        end
      rescue Redis::BaseConnectionError
        # ignore
      end

      def tripped_key(domain)
        "http_cb_tripped_#{domain}"
      end

      def threshold_key(domain)
        "http_cb_counter_#{domain}"
      end

      def redis_client
        @redis.respond_to?(:call) ? @redis.call : @redis || nil
      end
    end
  end

View on GitHub (pinned to 1c9f0bb801)