What causes "API error %d: %s" in sipeed/picoclaw?

Reduce concurrency / add client-side throttling so 429s stop firing

API error %d: %s

Error message

API error %d: %s

What it means

Transient HTTP failure classification in membench's LLM client: any 429 or 5xx status is wrapped as 'API error <status>: <body>' and assigned to lastErr, then the loop continues with exponential backoff (1s, 2s, 4s...). This error object is normally never returned to the caller — it only escapes wrapped by 'after N retries' if the budget is exhausted (llm_client.go:164-166).

Source

Thrown at cmd/membench/llm_client.go:164

			if c.APIKey != "" {
				req.Header.Set("Authorization", "Bearer "+c.APIKey)
			}
		}

		var resp *http.Response
		resp, lastErr = c.Client.Do(req)
		if lastErr != nil {
			continue // network/timeout error → retry
		}

		respBody, lastErr = io.ReadAll(resp.Body)
		_ = resp.Body.Close()
		if lastErr != nil {
			continue
		}

		if resp.StatusCode == 429 || resp.StatusCode >= 500 {
			lastErr = fmt.Errorf("API error %d: %s", resp.StatusCode, string(respBody))
			continue // rate limit or server error → retry
		}
		if resp.StatusCode != 200 {
			return "", fmt.Errorf("API error %d: %s", resp.StatusCode, string(respBody))
		}

		lastErr = nil
		break
	}
	if lastErr != nil {
		return "", fmt.Errorf("after %d retries: %w", c.MaxRetries, lastErr)
	}

	var chatResp chatResponse
	if err := json.Unmarshal(respBody, &chatResp); err != nil {
		return "", fmt.Errorf("parse response: %w", err)
	}
	if len(chatResp.Choices) == 0 {

View on GitHub (pinned to 49183d7e8d)

Solutions

Reduce concurrency / add client-side throttling so 429s stop firing
Increase MaxRetries on the client so the backoff schedule can outlast the rate-limit window
Read the response body — provider JSON usually names the real cause (quota, model loading)
Honor Retry-After when present instead of the fixed 1s/2s/4s ladder

Example fix

// before
backoff := time.Duration(1<<(attempt-1)) * time.Second

// after
backoff := time.Duration(1<<(attempt-1)) * time.Second
if ra := resp.Header.Get("Retry-After"); ra != "" {
    if secs, err := strconv.Atoi(ra); err == nil {
        backoff = time.Duration(secs) * time.Second
    }
}

Defensive patterns

Strategy: retry

Validate before calling

// probe the provider's rate posture before a benchmark run
resp, err := http.Get(baseURL + "/models")
if err == nil && resp.StatusCode == 429 {
    return fmt.Errorf("already rate-limited at %s; delay the run", baseURL)
}

Try / catch

// already inside the client: keep 429/5xx on lastErr and continue
if resp.StatusCode == 429 || resp.StatusCode >= 500 {
    lastErr = fmt.Errorf("API error %d: %s", resp.StatusCode, string(respBody))
    continue
}
// caller side: this error only surfaces wrapped by 'after N retries' — handle it there

Prevention

Throttle concurrency below the provider's TPM/RPM limits
Honor Retry-After when present instead of the fixed 1s/2s/4s ladder
Size MaxRetries so total backoff exceeds the throttling window
Watch for 503 from Ollama while a model loads — warm it up with one request before benchmarking

When it happens

Trigger: Provider rate limiting (429) under concurrent eval workers; OpenAI-compatible server 500/502/503 during load or model cold-start; Ollama returning 503 while the model is loading into memory; proxy or gateway 5xx.

Common situations: High-parallelism benchmark runs tripping tokens-per-minute limits; self-hosted inference server overloaded; maintenance windows on hosted APIs; retries firing against a 429 whose Retry-After exceeds the fixed backoff schedule.

Related errors

AI-assisted analysis of sipeed/picoclaw@49183d7e8d (2026-08-15). Data as JSON: /api/errors/5d8458f387ee6f60. Report an issue: GitHub.