{"record":{"id":"cfbb11a390cec043","repo":"siyuan-note/siyuan","slug":"response-too-large-cfbb11","errorCode":null,"errorMessage":"response too large","messagePattern":"response too large","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"kernel/util/webfetch.go","lineNumber":70,"sourceCode":"\t}\n\n\tresp, err := httpclient.NewBrowserRequest().Get(rawURL)\n\tif err != nil {\n\t\treturn \"\", errors.New(\"fetch failed: \" + err.Error())\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode >= 400 {\n\t\treturn \"\", fmt.Errorf(\"HTTP %d\", resp.StatusCode)\n\t}\n\n\tcontentType := resp.Header.Get(\"Content-Type\")\n\tmaxReadBytes := int64(maxWebFetchBytes)\n\tif !strings.HasPrefix(contentType, \"text/html\") && !strings.HasPrefix(contentType, \"text/plain\") {\n\t\tmaxReadBytes = maxWebFetchFileBytes\n\t}\n\tif resp.ContentLength > maxReadBytes {\n\t\treturn \"\", errors.New(\"response too large\")\n\t}\n\n\tbody, err := io.ReadAll(io.LimitReader(resp.Body, maxReadBytes))\n\tif err != nil {\n\t\treturn \"\", errors.New(\"read body failed: \" + err.Error())\n\t}\n\n\tif !strings.HasPrefix(contentType, \"text/html\") && !strings.HasPrefix(contentType, \"text/plain\") {\n\t\timportDir := filepath.Join(TempDir, \"import\")\n\t\tif merr := os.MkdirAll(importDir, 0755); merr != nil {\n\t\t\treturn \"\", errors.New(\"create import dir failed: \" + merr.Error())\n\t\t}\n\t\tfilename := extractFilename(rawURL, contentType)\n\t\tfilePath := filepath.Join(importDir, filename)\n\t\tif werr := os.WriteFile(filePath, body, 0644); werr != nil {\n\t\t\treturn \"\", errors.New(\"write file failed: \" + werr.Error())\n\t\t}\n\t\treturn fmt.Sprintf(\"Saved to: %s (%d bytes)\", filePath, len(body)), nil","sourceCodeStart":52,"sourceCodeEnd":88,"githubUrl":"https://github.com/siyuan-note/siyuan/blob/251596fc0de2f9528c00c224252fd073a99973f4/kernel/util/webfetch.go#L52-L88","documentation":"Returned at webfetch.go:70 when resp.ContentLength exceeds the cap declared in webfetch.go:36-39: 5 MiB for text/html or text/plain, 10 MiB for any other content type. Note the check uses the server-declared Content-Length header, so chunked/streaming responses (ContentLength == -1) skip this guard and are instead capped by io.LimitReader at read time.","triggerScenarios":"Pointing WebFetch at a large PDF/image/archive whose declared Content-Length exceeds 10 MiB, or an HTML/plain page larger than 5 MiB.","commonSituations":"User pastes a direct media-asset URL expecting page text; fetching a large dataset dump; CDN reporting true Content-Length on a big file.","solutions":["Fetch a smaller URL — WebFetch is meant for pages/text, not large downloads.","Download the large file out-of-band (browser, wget) and import it separately.","If you genuinely need it, raise maxWebFetchFileBytes in kernel/util/webfetch.go and rebuild the kernel (local change only)."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// HEAD the URL to learn the declared size before paying for the GET.\nfunc declaredTooLarge(rawURL string) (bool, error) {\n    resp, err := httpclient.NewBrowserRequest().Head(rawURL)\n    if err != nil {\n        return false, err\n    }\n    defer resp.Body.Close()\n    return resp.ContentLength > int64(10*1024*1024), nil\n}","typeGuard":"func isResponseTooLarge(err error) bool {\n    return err != nil && err.Error() == \"response too large\"\n}","tryCatchPattern":"out, err := util.WebFetch(raw, \"markdown\")\nif err != nil && isResponseTooLarge(err) {\n    // surface a user action: pick a smaller URL or download out-of-band\n    return errors.New(\"target is too large to fetch via WebFetch; download it directly\")\n}","preventionTips":["Reserve WebFetch for HTML/text pages, not large assets.","HEAD-check Content-Length for untrusted URLs before fetching.","Remember chunked responses (no Content-Length) bypass this guard and are capped at read time instead."],"tags":["limits","http","fetch","content-length"],"backgroundTag":null,"analyzedSha":"251596fc0de2f9528c00c224252fd073a99973f4","analyzedAt":"2026-08-12T21:18:37.123Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}