amir20/dozzle · warning

cloud dispatcher rate limited, retry after

Error message

cloud dispatcher rate limited, retry after %s

What it means

CloudDispatcher.Send checks an internal circuit breaker (blockedUntil atomic timestamp) before dialing Dozzle Cloud. When a previous 429 or auth rejection tripped the breaker and now is still inside the backoff window, Send skips the HTTP request entirely and returns this error carrying the RFC3339 time until which all sends are blocked. It is a deliberate client-side rate limiter, not a remote failure at send time.

Solutions

1. Wait until the retry-after timestamp in the error message before sending again
2. Fix the root cause: verify the API key is valid; if a status check succeeds, call ResetBreaker() to clear blockedUntil
3. Reduce notification volume or debounce rules so 429s stop re-tripping the breaker
4. Recreate the dispatcher with a fresh valid API key, which resets the breaker state

Example fix

// before
err := dispatcher.Send(ctx, n) // fails while breaker is open
// after
if err := dispatcher.Send(ctx, n); err != nil && strings.Contains(err.Error(), "retry after") {
    // parse timestamp, retry after that time, or check key validity and call ResetBreaker()
}

if err := d.Send(ctx, n); err != nil {
    if strings.HasPrefix(err.Error(), "cloud dispatcher rate limited") {
        // parse RFC3339 timestamp from message; schedule retry after it
    }
}

When it happens

Trigger: Calling CloudDispatcher.Send after an earlier send received HTTP 429 (backoff set, default 60s or Retry-After) or 401/403 (backoff set to 6h), while time.Now() is still before blockedUntil.

Common situations: Frequent alert bursts hitting Dozzle Cloud rate limits; an expired or invalid DOLIGENCE API key that tripped the 6h breaker on the first 401/403; notification rules firing in a loop.

Related errors

• cloud rate limited, backing off for
• Failed to save destination
• dispatchers fetch failed
• rule POST failed
• webhook notification failed

AI-assisted analysis of amir20/dozzle@d9463cbe21 (2026-09-07). Data as JSON: /api/errors/6b52c05043e129e7. Report an issue: GitHub.

Appendix: source

Thrown at internal/notification/dispatcher/cloud.go:75

// API key). Retrying won't help until the user fixes their key, which recreates the
// dispatcher and resets the breaker.
const unauthorizedRetryAfter = 6 * time.Hour

// ResetBreaker clears the circuit breaker so the next Send dials cloud again.
// Called when a cloud status check succeeds, proving the API key is valid.
func (c *CloudDispatcher) ResetBreaker() {
	c.blockedUntil.Store(0)
}

// Send sends a notification to Dozzle Cloud
func (c *CloudDispatcher) Send(ctx context.Context, notification types.Notification) error {
	if blockedUntil := c.blockedUntil.Load(); blockedUntil > 0 && time.Now().UnixNano() < blockedUntil {
		t := time.Unix(0, blockedUntil)
		log.Debug().
			Str("cloud", c.Name).
			Time("blocked_until", t).
			Msg("circuit breaker open, skipping cloud request")
		return fmt.Errorf("cloud dispatcher rate limited, retry after %s", t.Format(time.RFC3339))
	}

	payload, err := json.Marshal(notification)
	if err != nil {
		return fmt.Errorf("failed to marshal notification: %w", err)
	}

	req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.URL, bytes.NewReader(payload))
	if err != nil {
		return fmt.Errorf("failed to create request: %w", err)
	}

	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("User-Agent", UserAgent)
	req.Header.Set("X-API-Key", c.APIKey)

	resp, err := c.client.Do(req)
	if err != nil {

View on GitHub (pinned to d9463cbe21)