What causes "service unavailable due to rate limit exceeded" in go-kratos/kratos?

Client side: honor the 429, back off and retry with jitter instead of hammering

go-kratos/kratos · error · github.com/go-kratos/kratos/v3/errors.Error

RATELIMIT

RATELIMIT

Error message

service unavailable due to rate limit exceeded

What it means

ErrLimitExceed is a typed kratos error (HTTP 429, reason RATELIMIT) defined in contrib/polaris/ratelimit.go:18 and returned by the polaris Ratelimit middleware. Inside the middleware, l.Allow(operation, args...) is called with the operation plus header/query arguments built from the request; if Allow returns an error (polaris quota denied), the middleware aborts the handler and returns ErrLimitExceed without executing business logic.

Source

Thrown at contrib/polaris/ratelimit.go:18

package polaris

import (
	"context"
	"strings"

	"github.com/go-kratos/kratos/v3/errors"
	"github.com/go-kratos/kratos/v3/middleware"
	"github.com/go-kratos/kratos/v3/middleware/ratelimit"
	"github.com/go-kratos/kratos/v3/transport"
	"github.com/go-kratos/kratos/v3/transport/http"

	"github.com/polarismesh/polaris-go/pkg/model"
)

// ErrLimitExceed is service unavailable due to rate limit exceeded.
var (
	ErrLimitExceed = errors.New(429, "RATELIMIT", "service unavailable due to rate limit exceeded")
)

// Ratelimit Request rate limit middleware
func Ratelimit(l Limiter) middleware.Middleware {
	return func(handler middleware.Handler) middleware.Handler {
		return func(ctx context.Context, req any) (reply any, err error) {
			if tr, ok := transport.FromServerContext(ctx); ok {
				var args []model.Argument
				headers := tr.RequestHeader()
				// handle header
				for _, header := range headers.Keys() {
					args = append(args, model.BuildHeaderArgument(header, headers.Get(header)))
				}
				// handle http
				if ht, ok := tr.(*http.Transport); ok {
					// url query
					for key, values := range ht.Request().URL.Query() {
						args = append(args, model.BuildQueryArgument(key, strings.Join(values, ",")))

View on GitHub (pinned to 668db92c2c)

Solutions

Client side: honor the 429, back off and retry with jitter instead of hammering
Check the polaris console rate-limit rules for the operation and adjust the quota if the limit is genuinely too low
Verify polaris server connectivity from the app - Allow errors on infra failure look identical to real rejections
Match precisely with kratos errors: if e := errors.FromError(err); e.Reason == "RATELIMIT" && e.Code == 429

Example fix

// client-side handling
reply, err := client.Call(ctx, req)
if e := kerrors.FromError(err); e.Code == 429 && e.Reason == "RATELIMIT" {
    time.Sleep(backoff()) // exponential + jitter, then retry
    reply, err = client.Call(ctx, req)
}

Defensive patterns

Strategy: retry

Try / catch

if reply, err = handler(ctx, req); err != nil {
    if e := kerrors.FromError(err); e.Code == 429 && e.Reason == "RATELIMIT" {
        // quota denied: back off and retry with jitter; do not retry immediately
    }
}

Prevention

Size polaris quota rules for peak traffic plus headroom before enabling the middleware
Differentiate infra failure from real rejection by monitoring polaris server health - Allow errors on both map to ErrLimitExceed
Clients: implement exponential backoff with jitter and honor Retry-After style guidance
Test rate rules in a staging namespace before applying to production

When it happens

Trigger: Attaching polaris.Ratelimit(l) as server middleware and receiving traffic that exceeds the quota rule configured in polaris for that operation/route; Allow also returns an error for any polaris quota API failure, so a broken polaris server connection can surface the same error. Client sees a 429 whose body encodes reason RATELIMIT.

Common situations: Load test or traffic spike tripping a polaris rate rule; rule configured per-header/per-query argument (the middleware forwards all headers and URL queries as model.Argument) so a specific caller is throttled; polaris server unreachable so every Allow fails and all requests appear rate-limited.

Understand the failure class

HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

AI-assisted analysis of go-kratos/kratos@668db92c2c (2026-08-16). Data as JSON: /api/errors/ee423471703bbc5a. Report an issue: GitHub.