{"record":{"id":"5c0b3a2e739e51ee","repo":"charmbracelet/crush","slug":"failed-to-fetch-url-w-5c0b3a","errorCode":null,"errorMessage":"failed to fetch URL: %w","messagePattern":"failed to fetch URL: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/agent/tools/fetch_helpers.go","lineNumber":38,"sourceCode":"const BrowserUserAgent = \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36\"\n\nvar multipleNewlinesRe = regexp.MustCompile(`\\n{3,}`)\n\n// FetchURLAndConvert fetches a URL and converts HTML content to markdown.\nfunc FetchURLAndConvert(ctx context.Context, client *http.Client, url string) (string, error) {\n\treq, err := http.NewRequestWithContext(ctx, \"GET\", url, nil)\n\tif err != nil {\n\t\treturn \"\", fmt.Errorf(\"failed to create request: %w\", err)\n\t}\n\n\t// Use realistic browser headers for better compatibility.\n\treq.Header.Set(\"User-Agent\", BrowserUserAgent)\n\treq.Header.Set(\"Accept\", \"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\")\n\treq.Header.Set(\"Accept-Language\", \"en-US,en;q=0.5\")\n\n\tresp, err := client.Do(req)\n\tif err != nil {\n\t\treturn \"\", fmt.Errorf(\"failed to fetch URL: %w\", err)\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode != http.StatusOK {\n\t\treturn \"\", fmt.Errorf(\"request failed with status code: %d\", resp.StatusCode)\n\t}\n\n\tmaxSize := int64(5 * 1024 * 1024) // 5MB\n\tbody, err := io.ReadAll(io.LimitReader(resp.Body, maxSize))\n\tif err != nil {\n\t\treturn \"\", fmt.Errorf(\"failed to read response body: %w\", err)\n\t}\n\n\tcontent := string(body)\n\n\tif !utf8.ValidString(content) {\n\t\treturn \"\", errors.New(\"response content is not valid UTF-8\")\n\t}","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/charmbracelet/crush/blob/7944b8e52225d8805e31eacbf7ef24856b0dfb7a/internal/agent/tools/fetch_helpers.go#L20-L56","documentation":"Wraps any transport-level failure from client.Do inside FetchURLAndConvert — the request was built successfully but never completed. DNS failures, refused connections, timeouts, and TLS errors all land here, with the original error preserved via %w.","triggerScenarios":"client.Do(req) returns non-nil error during FetchURLAndConvert: unreachable host, connection timeout, TLS handshake failure, or request context cancellation (ctx done) mid-flight.","commonSituations":"Fetching URLs from an offline or sandboxed environment, corporate proxies blocking egress, slow hosts exceeding the client timeout, or the tool's context being cancelled by the user/agent.","solutions":["Test the URL with curl from the same environment to confirm reachability","Inspect the wrapped error for DNS vs TLS vs timeout cause","Increase the http.Client timeout if legitimate slow sites are timing out","Check proxy settings (HTTP_PROXY/HTTPS_PROXY) and firewall egress rules"],"exampleFix":"// before\nresp, err := client.Do(req)\nif err != nil {\n    return \"\", fmt.Errorf(\"failed to fetch URL: %w\", err)\n}\n// after\nresp, err := client.Do(req)\nif err != nil {\n    if errors.Is(err, context.DeadlineExceeded) {\n        return \"\", fmt.Errorf(\"fetch timed out: %w\", err)\n    }\n    return \"\", fmt.Errorf(\"failed to fetch URL: %w\", err)\n}","handlingStrategy":"retry","validationCode":"u, err := url.Parse(rawURL)\nif err != nil || u.Host == \"\" {\n    return fmt.Errorf(\"invalid URL: %q\", rawURL)\n}\n// optionally: net.DialTimeout(\"tcp\", host, 3*time.Second) reachability probe","typeGuard":"var netErr net.Error\nif errors.As(err, &netErr) && netErr.Timeout() {\n    // safe to retry\n}","tryCatchPattern":"resp, err := client.Do(req)\nif err != nil {\n    if errors.Is(err, context.Canceled) { return \"\", err }\n    return \"\", fmt.Errorf(\"failed to fetch URL: %w\", err) // retry on timeout/reset only\n}","preventionTips":["Set explicit client.Timeout so hangs become clear errors","Retry transient errors (timeout, connection reset) with backoff","Check proxy/firewall egress rules in CI and containers","Cancel-aware: distinguish context.Canceled from real network failures"],"tags":["network","http","timeout","dns"],"backgroundTag":"http-request-failed","analyzedSha":"7944b8e52225d8805e31eacbf7ef24856b0dfb7a","analyzedAt":"2026-08-29T12:48:59.079Z","schemaVersion":2},"datasetVersion":"2026-08-29T17:17:51.833Z"}