charmbracelet/crush · error

failed to create request: %w

Error message

failed to create request: %w

What it means

http.NewRequestWithContext failed to parse the given URL into a valid HTTP GET request. This library surfaces the parse error wrapped as 'failed to create request'. It is thrown before any network I/O occurs, so it is purely a client-side input validation failure.

Source

Thrown at internal/agent/tools/fetch_helpers.go:28

	"net/http"
	"regexp"
	"strings"
	"unicode/utf8"

	md "github.com/JohannesKaufmann/html-to-markdown"
	"golang.org/x/net/html"
)

// BrowserUserAgent is a realistic browser User-Agent for better compatibility.
const BrowserUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"

var multipleNewlinesRe = regexp.MustCompile(`\n{3,}`)

// FetchURLAndConvert fetches a URL and converts HTML content to markdown.
func FetchURLAndConvert(ctx context.Context, client *http.Client, url string) (string, error) {
	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
	if err != nil {
		return "", fmt.Errorf("failed to create request: %w", err)
	}

	// Use realistic browser headers for better compatibility.
	req.Header.Set("User-Agent", BrowserUserAgent)
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
	req.Header.Set("Accept-Language", "en-US,en;q=0.5")

	resp, err := client.Do(req)
	if err != nil {
		return "", fmt.Errorf("failed to fetch URL: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return "", fmt.Errorf("request failed with status code: %d", resp.StatusCode)
	}

	maxSize := int64(5 * 1024 * 1024) // 5MB

View on GitHub (pinned to 7944b8e522)

Solutions

1. Validate the URL with url.ParseRequestURI before calling and require http:// or https:// scheme
2. Trim whitespace and reject/pre-encode URLs containing spaces or control characters
3. Return a clearer tool error to the model telling it the URL format is invalid
4. Prepend https:// if the scheme is missing, when appropriate

Example fix

// before
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
    return "", fmt.Errorf("failed to create request: %w", err)
}
// after
u, perr := url.Parse(strings.TrimSpace(url))
if perr != nil || (u.Scheme != "http" && u.Scheme != "https") {
    return "", fmt.Errorf("invalid or unsupported URL: %q", url)
}
req, err := http.NewRequestWithContext(ctx, "GET", u.String(), nil)
if err != nil {
    return "", fmt.Errorf("failed to create request: %w", err)
}

u, err := url.ParseRequestURI(strings.TrimSpace(rawURL))
if err != nil || (u.Scheme != "http" && u.Scheme != "https") {
    return fmt.Errorf("URL must be absolute http(s): %q", rawURL)
}

null

req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
    return "", fmt.Errorf("failed to create request: %w", err)
}

When it happens

Trigger: FetchURLAndConvert (and thus the fetch tool) is called with a URL that has an unsupported scheme (not http/https, e.g. 'ftp://'), contains invalid characters (spaces, unescaped control chars), or is otherwise unparsable by net/url.

Common situations: The LLM passes a URL without a scheme ('example.com'), with embedded spaces, or a file:// scheme; malformed percent-encoding; user-supplied URLs from chat input not validated upstream.

Related errors

• failed to create request for provider %s: %w
• failed to create request: %w
• failed to create request: %w
• mcp http config requires a non-empty 'url' field
• could not create request: %w

AI-assisted analysis of charmbracelet/crush@7944b8e522 (2026-08-29). Data as JSON: /api/errors/de53322fa115c3a7. Report an issue: GitHub.