{"record":{"id":"cfbb11a390cec043","repo":"siyuan-note/siyuan","slug":"response-too-large-cfbb11","errorCode":null,"errorMessage":"response too large","messagePattern":"response too large","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"kernel/util/webfetch.go","lineNumber":69,"sourceCode":"\t}\n\n\tresp, err := ssrfSafeClient.Get(rawURL)\n\tif err != nil {\n\t\treturn \"\", errors.New(\"fetch failed: \" + err.Error())\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode >= 400 {\n\t\treturn \"\", fmt.Errorf(\"HTTP %d\", resp.StatusCode)\n\t}\n\n\tcontentType := resp.Header.Get(\"Content-Type\")\n\tmaxReadBytes := int64(maxWebFetchBytes)\n\tif !strings.HasPrefix(contentType, \"text/html\") && !strings.HasPrefix(contentType, \"text/plain\") {\n\t\tmaxReadBytes = maxWebFetchFileBytes\n\t}\n\tif resp.ContentLength > maxReadBytes {\n\t\treturn \"\", errors.New(\"response too large\")\n\t}\n\n\tbody, err := io.ReadAll(io.LimitReader(resp.Body, maxReadBytes))\n\tif err != nil {\n\t\treturn \"\", errors.New(\"read body failed: \" + err.Error())\n\t}\n\n\tif !strings.HasPrefix(contentType, \"text/html\") && !strings.HasPrefix(contentType, \"text/plain\") {\n\t\timportDir := filepath.Join(TempDir, \"import\")\n\t\tif merr := os.MkdirAll(importDir, 0755); merr != nil {\n\t\t\treturn \"\", errors.New(\"create import dir failed: \" + merr.Error())\n\t\t}\n\t\tfilename := extractFilename(rawURL, contentType)\n\t\tfilePath := filepath.Join(importDir, filename)\n\t\tif werr := os.WriteFile(filePath, body, 0644); werr != nil {\n\t\t\treturn \"\", errors.New(\"write file failed: \" + werr.Error())\n\t\t}\n\t\treturn fmt.Sprintf(\"Saved to: %s (%d bytes)\", filePath, len(body)), nil","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/siyuan-note/siyuan/blob/9f775e8a12daef8255556097396f9b2739078892/kernel/util/webfetch.go#L51-L87","documentation":"WebFetch enforces a size cap before reading the body: if resp.ContentLength exceeds maxReadBytes (maxWebFetchBytes for HTML/text content types, the smaller maxWebFetchFileBytes otherwise), it aborts with 'response too large'. The limit is declared via Content-Length, so chunked responses without a length are only partially read instead of rejected here.","triggerScenarios":"Fetching a very large HTML page or a non-HTML file (binary, PDF, archive) whose Content-Length exceeds the configured byte limit.","commonSituations":"Pointing WebFetch at large downloadable files instead of pages; fetching huge generated HTML dashboards; a site serving a multi-hundred-MB payload where a small file was expected.","solutions":["Fetch a smaller resource or a more specific URL (direct page, not a file dump)","Check the resource's Content-Length with a HEAD request before calling WebFetch","If the file is legitimately needed, download it through a different path that streams to disk instead of WebFetch","Increase the maxWebFetchBytes/maxWebFetchFileBytes limits only if memory budgets allow"],"exampleFix":"// before\n_, err := WebFetch(\"https://example.com/big-dump.zip\", \"markdown\")\n\n// after\nresp, _ := http.Head(\"https://example.com/big-dump.zip\")\nif resp.ContentLength <= 5*1024*1024 {\n    _, err = WebFetch(\"https://example.com/big-dump.zip\", \"markdown\")\n} else {\n    err = errors.New(\"use streaming download instead\")\n}","handlingStrategy":"validation","validationCode":"resp, err := http.Head(url)\nif err == nil && resp.ContentLength > maxWebFetchBytes {\n    return errors.New(\"skipping: payload exceeds WebFetch limit; use streaming download\")\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["HEAD-check Content-Length for large or unknown resources","Use streaming download-to-disk for big files instead of WebFetch","Remember non-HTML/text types get the smaller maxWebFetchFileBytes cap","Budget memory: WebFetch buffers the whole body in RAM"],"tags":["http","size-limit","payload"],"backgroundTag":"payload-too-large","analyzedSha":"9f775e8a12daef8255556097396f9b2739078892","analyzedAt":"2026-09-19T03:17:15.984Z","contentChangedAt":"2026-09-19T03:17:15.984Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}