charmbracelet/crush · error

failed to fetch URL: %w

Error message

failed to fetch URL: %w

What it means

Wraps any transport-level failure from client.Do inside FetchURLAndConvert — the request was built successfully but never completed. DNS failures, refused connections, timeouts, and TLS errors all land here, with the original error preserved via %w.

Source

Thrown at internal/agent/tools/fetch_helpers.go:38

const BrowserUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"

var multipleNewlinesRe = regexp.MustCompile(`\n{3,}`)

// FetchURLAndConvert fetches a URL and converts HTML content to markdown.
func FetchURLAndConvert(ctx context.Context, client *http.Client, url string) (string, error) {
	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
	if err != nil {
		return "", fmt.Errorf("failed to create request: %w", err)
	}

	// Use realistic browser headers for better compatibility.
	req.Header.Set("User-Agent", BrowserUserAgent)
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
	req.Header.Set("Accept-Language", "en-US,en;q=0.5")

	resp, err := client.Do(req)
	if err != nil {
		return "", fmt.Errorf("failed to fetch URL: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return "", fmt.Errorf("request failed with status code: %d", resp.StatusCode)
	}

	maxSize := int64(5 * 1024 * 1024) // 5MB
	body, err := io.ReadAll(io.LimitReader(resp.Body, maxSize))
	if err != nil {
		return "", fmt.Errorf("failed to read response body: %w", err)
	}

	content := string(body)

	if !utf8.ValidString(content) {
		return "", errors.New("response content is not valid UTF-8")
	}

View on GitHub (pinned to 7944b8e522)

Solutions

1. Test the URL with curl from the same environment to confirm reachability
2. Inspect the wrapped error for DNS vs TLS vs timeout cause
3. Increase the http.Client timeout if legitimate slow sites are timing out
4. Check proxy settings (HTTP_PROXY/HTTPS_PROXY) and firewall egress rules

Example fix

// before
resp, err := client.Do(req)
if err != nil {
    return "", fmt.Errorf("failed to fetch URL: %w", err)
}
// after
resp, err := client.Do(req)
if err != nil {
    if errors.Is(err, context.DeadlineExceeded) {
        return "", fmt.Errorf("fetch timed out: %w", err)
    }
    return "", fmt.Errorf("failed to fetch URL: %w", err)
}

u, err := url.Parse(rawURL)
if err != nil || u.Host == "" {
    return fmt.Errorf("invalid URL: %q", rawURL)
}
// optionally: net.DialTimeout("tcp", host, 3*time.Second) reachability probe

var netErr net.Error
if errors.As(err, &netErr) && netErr.Timeout() {
    // safe to retry
}

resp, err := client.Do(req)
if err != nil {
    if errors.Is(err, context.Canceled) { return "", err }
    return "", fmt.Errorf("failed to fetch URL: %w", err) // retry on timeout/reset only
}

When it happens

Trigger: client.Do(req) returns non-nil error during FetchURLAndConvert: unreachable host, connection timeout, TLS handshake failure, or request context cancellation (ctx done) mid-flight.

Common situations: Fetching URLs from an offline or sandboxed environment, corporate proxies blocking egress, slow hosts exceeding the client timeout, or the tool's context being cancelled by the user/agent.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• failed to fetch URL: %w
• failed to fetch URL: %w
• failed to download from URL: %w
• failed to read response body: %w
• failed to make request: %w

AI-assisted analysis of charmbracelet/crush@7944b8e522 (2026-08-29). Data as JSON: /api/errors/5c0b3a2e739e51ee. Report an issue: GitHub.