{"record":{"id":"c9ca0fc383b2d21d","repo":"siyuan-note/siyuan","slug":"response-too-large","errorCode":null,"errorMessage":"response too large","messagePattern":"response too large","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"kernel/util/httprequest.go","lineNumber":351,"sourceCode":"\tresp, err := ssrfSafeClient.Do(req)\n\tif err != nil {\n\t\treturn 0, \"\", \"\", errors.New(\"request failed: \" + err.Error())\n\t}\n\tif resp == nil {\n\t\treturn 0, \"\", \"\", errors.New(\"nil response\")\n\t}\n\tdefer resp.Body.Close()\n\n\tstatusCode = resp.StatusCode\n\tcontentType = resp.Header.Get(\"Content-Type\")\n\n\tmaxReadBytes := int64(maxHTTPRequestBytes)\n\tif !isTextContentType(contentType) {\n\t\tmaxReadBytes = maxHTTPRequestFileBytes\n\t}\n\t// ContentLength 为 -1（chunked）时跳过大小预检，交由 LimitReader 兜底截断。\n\tif resp.ContentLength > maxReadBytes {\n\t\treturn statusCode, contentType, \"\", errors.New(\"response too large\")\n\t}\n\n\trespBody, rerr := io.ReadAll(io.LimitReader(resp.Body, maxReadBytes))\n\tif rerr != nil {\n\t\treturn statusCode, contentType, \"\", errors.New(\"read body failed: \" + rerr.Error())\n\t}\n\n\t// 二进制响应落盘，返回文件路径，供智能体按需进一步处理。\n\tif !isTextContentType(contentType) {\n\t\timportDir := filepath.Join(TempDir, \"import\")\n\t\tif merr := os.MkdirAll(importDir, 0755); merr != nil {\n\t\t\treturn statusCode, contentType, \"\", errors.New(\"create import dir failed: \" + merr.Error())\n\t\t}\n\t\tfilename := extractFilename(rawURL, contentType)\n\t\tfilePath := filepath.Join(importDir, filename)\n\t\tif werr := os.WriteFile(filePath, respBody, 0644); werr != nil {\n\t\t\treturn statusCode, contentType, \"\", errors.New(\"write file failed: \" + werr.Error())\n\t\t}","sourceCodeStart":333,"sourceCodeEnd":369,"githubUrl":"https://github.com/siyuan-note/siyuan/blob/9f775e8a12daef8255556097396f9b2739078892/kernel/util/httprequest.go#L333-L369","documentation":"HTTPRequest enforces a size cap before reading the body: maxHTTPRequestBytes for text content types and the larger maxHTTPRequestFileBytes for non-text types. If the response's declared ContentLength exceeds the applicable cap, it refuses to read and returns \"response too large\" to avoid exhausting memory or disk on huge downloads.","triggerScenarios":"Calling HTTPRequest against an endpoint whose Content-Length header exceeds the cap — e.g. a large file download served with a text content type (capped at maxHTTPRequestBytes), or any declared size above the binary cap; note chunked responses (ContentLength -1) skip this precheck and are instead truncated by the LimitReader.","commonSituations":"Fetching big media/archive URLs that declare a text/* or missing Content-Type; a server mislabeling a large binary as text/plain; pointing the agent fetch at a dataset or backup file far larger than intended; a redirect chain ending on a multi-GB asset.","solutions":["Fetch a smaller resource or a range/partial URL instead of the whole file","Ensure the server sends the correct Content-Type so the larger file cap applies to genuinely binary resources","Download large files out-of-band (directly with curl/wget) rather than through HTTPRequest","If the cap is genuinely too small for your deployment, adjust maxHTTPRequestBytes/maxHTTPRequestFileBytes in kernel/util/httprequest.go consciously, accepting the memory cost"],"exampleFix":"// before\nutil.HTTPRequest(ctx, \"https://example.com/dump.sql\") // Content-Type text/plain, 2 GB\n// after\nutil.HTTPRequest(ctx, \"https://example.com/dump.sql.meta\") // small manifest instead","handlingStrategy":"validation","validationCode":"head, _ := http.Head(target)\nsize := head.ContentLength\nct := head.Header.Get(\"Content-Type\")\ncap := int64(64 << 20)\nif !isTextLike(ct) && size > cap { return fmt.Errorf(\"skipping oversized %s (%d bytes)\", target, size) }","typeGuard":null,"tryCatchPattern":"if err != nil && err.Error() == \"response too large\" {\n    // skip the resource or fetch a partial/range URL instead\n}","preventionTips":["HEAD the target and check Content-Length first","Skip or stream large resources out-of-band","Keep server Content-Type headers accurate so binary caps apply"],"tags":["http","size-limit","payload","go"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"9f775e8a12daef8255556097396f9b2739078892","analyzedAt":"2026-09-19T03:17:15.984Z","contentChangedAt":"2026-09-19T03:17:15.984Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}