{"record":{"id":"3e56f6f56e550d58","repo":"siyuan-note/siyuan","slug":"http-d","errorCode":null,"errorMessage":"HTTP %d","messagePattern":"HTTP %d","errorType":"http","errorClass":null,"httpStatus":null,"severity":"error","filePath":"kernel/util/webfetch.go","lineNumber":60,"sourceCode":"\tif err != nil || (u.Scheme != \"http\" && u.Scheme != \"https\") {\n\t\treturn \"\", errors.New(\"URL must start with http:// or https://\")\n\t}\n\tif u.Host == \"\" {\n\t\treturn \"\", errors.New(\"URL has no host\")\n\t}\n\n\tif err := CheckHostSSRF(u.Hostname()); err != nil {\n\t\treturn \"\", err\n\t}\n\n\tresp, err := ssrfSafeClient.Get(rawURL)\n\tif err != nil {\n\t\treturn \"\", errors.New(\"fetch failed: \" + err.Error())\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode >= 400 {\n\t\treturn \"\", fmt.Errorf(\"HTTP %d\", resp.StatusCode)\n\t}\n\n\tcontentType := resp.Header.Get(\"Content-Type\")\n\tmaxReadBytes := int64(maxWebFetchBytes)\n\tif !strings.HasPrefix(contentType, \"text/html\") && !strings.HasPrefix(contentType, \"text/plain\") {\n\t\tmaxReadBytes = maxWebFetchFileBytes\n\t}\n\tif resp.ContentLength > maxReadBytes {\n\t\treturn \"\", errors.New(\"response too large\")\n\t}\n\n\tbody, err := io.ReadAll(io.LimitReader(resp.Body, maxReadBytes))\n\tif err != nil {\n\t\treturn \"\", errors.New(\"read body failed: \" + err.Error())\n\t}\n\n\tif !strings.HasPrefix(contentType, \"text/html\") && !strings.HasPrefix(contentType, \"text/plain\") {\n\t\timportDir := filepath.Join(TempDir, \"import\")","sourceCodeStart":42,"sourceCodeEnd":78,"githubUrl":"https://github.com/siyuan-note/siyuan/blob/9f775e8a12daef8255556097396f9b2739078892/kernel/util/webfetch.go#L42-L78","documentation":"WebFetch treats any HTTP status >= 400 as a failure and returns a compact 'HTTP <code>' error without the response body. It deliberately does not differentiate 4xx/5xx beyond the numeric code, so the caller must interpret the status themselves.","triggerScenarios":"The remote server returned 404 (page removed), 403 (bot blocking/paywall), 429 (rate limited), 500 (server error), or any other >= 400 status for the fetched URL.","commonSituations":"Scraping a URL that now redirects to a login page; fetching pages that block non-browser User-Agents; expired links in documents; APIs behind auth being fetched as public pages; hitting a site's rate limiter.","solutions":["Parse the status code from the 'HTTP <code>' message and handle 404/403/429 distinctly","For 403/429, the site likely blocks the client — use an official API or fetch with proper credentials elsewhere","Retry with backoff on 429/5xx; do not retry on 4xx client errors like 404","Verify the URL in a browser; follow redirects manually if the final target differs"],"exampleFix":"// before\ncontent, err := WebFetch(url, \"markdown\")\n\n// after\ncontent, err := WebFetch(url, \"markdown\")\nif err != nil {\n    var code int\n    if _, perr := fmt.Sscanf(err.Error(), \"HTTP %d\", &code); perr == nil && code == 429 {\n        time.Sleep(backoff) // retry rate-limited fetch\n    }\n}","handlingStrategy":"try-catch","validationCode":"resp, err := http.Head(url)\nif err == nil && resp.StatusCode >= 400 {\n    return fmt.Errorf(\"skipping: preflight status %d\", resp.StatusCode)\n}","typeGuard":null,"tryCatchPattern":"content, err := WebFetch(url, format)\nif err != nil {\n    var code int\n    if n, _ := fmt.Sscanf(err.Error(), \"HTTP %d\", &code); n == 1 {\n        switch {\n        case code == 429 || code >= 500: // retry with backoff\n        case code == 404:            // drop the link permanently\n        default:                     // log and skip\n        }\n    }\n}","preventionTips":["HEAD-probe links before batch fetching","Expect 403/429 from bot-protected sites and use official APIs where available","Classify retryable (429, 5xx) vs permanent (4xx) statuses by parsing the code","Keep link lists fresh; prune 404s"],"tags":["http","status-code","network"],"backgroundTag":"http-error-response","analyzedSha":"9f775e8a12daef8255556097396f9b2739078892","analyzedAt":"2026-09-19T03:17:15.984Z","contentChangedAt":"2026-09-19T03:17:15.984Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}