What causes "after %d retries: %w" in sipeed/picoclaw?

Unwrap the cause (errors.Unwrap) — fix that first; the retry layer only tells you it never succeeded

after %d retries: %w

Error message

after %d retries: %w

What it means

Retry budget exhausted in membench's LLM client: after MaxRetries re-attempts (initial try + N retries with 1s/2s/4s... backoff) the last error — a network/timeout failure, a 429, or a 5xx — is wrapped as 'after <N> retries: <cause>'. The %w wrap preserves the original error for errors.Is/As inspection, so the root cause is always the suffix of the message.

Source

Thrown at cmd/membench/llm_client.go:175

		respBody, lastErr = io.ReadAll(resp.Body)
		_ = resp.Body.Close()
		if lastErr != nil {
			continue
		}

		if resp.StatusCode == 429 || resp.StatusCode >= 500 {
			lastErr = fmt.Errorf("API error %d: %s", resp.StatusCode, string(respBody))
			continue // rate limit or server error → retry
		}
		if resp.StatusCode != 200 {
			return "", fmt.Errorf("API error %d: %s", resp.StatusCode, string(respBody))
		}

		lastErr = nil
		break
	}
	if lastErr != nil {
		return "", fmt.Errorf("after %d retries: %w", c.MaxRetries, lastErr)
	}

	var chatResp chatResponse
	if err := json.Unmarshal(respBody, &chatResp); err != nil {
		return "", fmt.Errorf("parse response: %w", err)
	}
	if len(chatResp.Choices) == 0 {
		return "", fmt.Errorf("no choices in response")
	}
	content := strings.TrimSpace(chatResp.Choices[0].Message.Content)
	// Strip any residual <think>...</think> blocks
	if idx := strings.Index(content, "</think>"); idx >= 0 {
		content = strings.TrimSpace(content[idx+len("</think>"):])
	}
	// Fallback: GLM/DeepSeek put thinking output in reasoning_content when thinking is enabled
	if content == "" && chatResp.Choices[0].Message.ReasoningContent != "" {
		content = strings.TrimSpace(chatResp.Choices[0].Message.ReasoningContent)
	}

View on GitHub (pinned to 49183d7e8d)

Solutions

Unwrap the cause (errors.Unwrap) — fix that first; the retry layer only tells you it never succeeded
Raise MaxRetries and/or the per-request timeout so the ladder outlasts throttling windows
Lower request rate/concurrency if the cause is 429
Check the endpoint is actually up (curl $BASE/models) before rerunning the full eval

Example fix

// before
if lastErr != nil {
    return "", fmt.Errorf("after %d retries: %w", c.MaxRetries, lastErr)
}

// after
if lastErr != nil {
    var statusErr *apiStatusError
    if errors.As(lastErr, &statusErr) && statusErr.Code == 429 {
        return "", fmt.Errorf("rate-limited after %d retries; lower concurrency or raise MaxRetries: %w", c.MaxRetries, lastErr)
    }
    return "", fmt.Errorf("after %d retries: %w", c.MaxRetries, lastErr)
}

Defensive patterns

Strategy: retry

Validate before calling

if err := preflightEndpoint(baseURL); err != nil {
    return err // avoid burning a long eval on a dead endpoint
}
// ensure MaxRetries and timeouts sized to worst-case throttling before the run
if c.MaxRetries < 5 { c.MaxRetries = 5 }

Try / catch

if err := llm.Complete(ctx, prompt); err != nil {
    if strings.Contains(err.Error(), "after") && strings.Contains(err.Error(), "retries") {
        cause := errors.Unwrap(err) // the real network/429/5xx failure
        if isTransient(cause) {
            time.Sleep(30 * time.Second) // let the throttle window reset
            return llm.Complete(ctx, prompt) // outer, coarse retry
        }
    }
    return err
}

Prevention

Always inspect errors.Unwrap of the 'after N retries' wrapper — the cause dictates the fix
Set per-request timeouts and a MaxRetries whose backoff ladder exceeds rate-limit windows
Reduce parallel workers when the cause is 429; retrying harder makes throttling worse
Persist partial eval progress so an exhausted retry budget doesn't restart the run

When it happens

Trigger: Sustained 429 rate limiting longer than the backoff ladder; provider outage spanning the whole retry window; dead endpoint where every Do() times out (server down, firewall drop); context deadline shorter than total backoff time.

Common situations: Benchmark runs against a quota that resets hourly; local model server crashed mid-eval; VPN/proxy dropping long-lived connections; MaxRetries left at default while the provider throttles aggressively.

Related errors

AI-assisted analysis of sipeed/picoclaw@49183d7e8d (2026-08-15). Data as JSON: /api/errors/1564bd5ec27a60b8. Report an issue: GitHub.