What causes "request failed with status code: %d" in charmbracelet/crush?

Verify the URL works in a browser or with `curl -I ` to see the real status

request failed with status code: %d

Error message

request failed with status code: %d

What it means

The server responded, but with a non-200 status code. FetchURLAndConvert treats anything other than HTTP 200 as an error and reports the numeric status. Unlike error 130/132 the request itself succeeded — the remote server rejected or redirected it without following to a 200.

Source

Thrown at internal/agent/tools/fetch_helpers.go:43

func FetchURLAndConvert(ctx context.Context, client *http.Client, url string) (string, error) {
	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
	if err != nil {
		return "", fmt.Errorf("failed to create request: %w", err)
	}

	// Use realistic browser headers for better compatibility.
	req.Header.Set("User-Agent", BrowserUserAgent)
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
	req.Header.Set("Accept-Language", "en-US,en;q=0.5")

	resp, err := client.Do(req)
	if err != nil {
		return "", fmt.Errorf("failed to fetch URL: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return "", fmt.Errorf("request failed with status code: %d", resp.StatusCode)
	}

	maxSize := int64(5 * 1024 * 1024) // 5MB
	body, err := io.ReadAll(io.LimitReader(resp.Body, maxSize))
	if err != nil {
		return "", fmt.Errorf("failed to read response body: %w", err)
	}

	content := string(body)

	if !utf8.ValidString(content) {
		return "", errors.New("response content is not valid UTF-8")
	}

	contentType := resp.Header.Get("Content-Type")

	// Convert HTML to markdown for better AI processing.
	if strings.Contains(contentType, "text/html") {

View on GitHub (pinned to 7944b8e522)

Solutions

Verify the URL works in a browser or with `curl -I <url>` to see the real status
Use a current, valid URL (404 means the page moved or never existed)
Check whether the site requires auth or API keys and use the appropriate authenticated endpoint
If bot-blocked (403 from CDN), the browser-like User-Agent already set may need cookies or a different fetch strategy

Example fix

// before
if resp.StatusCode != http.StatusOK {
    return "", fmt.Errorf("request failed with status code: %d", resp.StatusCode)
}
// after
if resp.StatusCode != http.StatusOK {
    return "", fmt.Errorf("request failed with status %d for %s", resp.StatusCode, url)
}

Defensive patterns

Strategy: type-guard

Validate before calling

resp, err := http.Head(url) // or curl -I
if err == nil && resp.StatusCode != http.StatusOK {
    return fmt.Errorf("URL returns status %d; not fetchable", resp.StatusCode)
}

Type guard

if resp.StatusCode != http.StatusOK {
    // handle 4xx/5xx explicitly before reading body
}

Try / catch

if resp.StatusCode != http.StatusOK {
    return "", fmt.Errorf("request failed with status code: %d", resp.StatusCode)
}

Prevention

Check status before parsing the body
Handle redirects and 404/410 by verifying the URL is current
Use authenticated endpoints when the resource requires auth
Distinguish 429 rate-limits (retry later) from 403 (do not retry)

When it happens

Trigger: The fetched URL returns 404 (missing page), 403 (blocked/bot-detected), 500 (server error), 301/302 loop, or 429 rate-limit — any status != http.StatusOK after redirects are followed.

Common situations: URLs that require authentication, sites blocking the client User-Agent, moved/removed pages, API endpoints requiring tokens, or Cloudflare bot protection returning 403.

Related errors

AI-assisted analysis of charmbracelet/crush@7944b8e522 (2026-08-29). Data as JSON: /api/errors/e95d57afaee050dc. Report an issue: GitHub.