siyuan-note/siyuan · error

response too large

Error message

response too large

What it means

HTTPRequest enforces a size cap before reading the body: maxHTTPRequestBytes for text content types and the larger maxHTTPRequestFileBytes for non-text types. If the response's declared ContentLength exceeds the applicable cap, it refuses to read and returns "response too large" to avoid exhausting memory or disk on huge downloads.

Solutions

1. Fetch a smaller resource or a range/partial URL instead of the whole file
2. Ensure the server sends the correct Content-Type so the larger file cap applies to genuinely binary resources
3. Download large files out-of-band (directly with curl/wget) rather than through HTTPRequest
4. If the cap is genuinely too small for your deployment, adjust maxHTTPRequestBytes/maxHTTPRequestFileBytes in kernel/util/httprequest.go consciously, accepting the memory cost

Example fix

// before
util.HTTPRequest(ctx, "https://example.com/dump.sql") // Content-Type text/plain, 2 GB
// after
util.HTTPRequest(ctx, "https://example.com/dump.sql.meta") // small manifest instead

head, _ := http.Head(target)
size := head.ContentLength
ct := head.Header.Get("Content-Type")
cap := int64(64 << 20)
if !isTextLike(ct) && size > cap { return fmt.Errorf("skipping oversized %s (%d bytes)", target, size) }

if err != nil && err.Error() == "response too large" {
    // skip the resource or fetch a partial/range URL instead
}

When it happens

Trigger: Calling HTTPRequest against an endpoint whose Content-Length header exceeds the cap — e.g. a large file download served with a text content type (capped at maxHTTPRequestBytes), or any declared size above the binary cap; note chunked responses (ContentLength -1) skip this precheck and are instead truncated by the LimitReader.

Common situations: Fetching big media/archive URLs that declare a text/* or missing Content-Type; a server mislabeling a large binary as text/plain; pointing the agent fetch at a dataset or backup file far larger than intended; a redirect chain ending on a multi-GB asset.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

• response too large
• custom emoji file is too large
• download failed:
• GitHub API returned status
• missing query param [u]

AI-assisted analysis of siyuan-note/siyuan@9f775e8a12 (2026-09-19). Data as JSON: /api/errors/c9ca0fc383b2d21d. Report an issue: GitHub.

Appendix: source

Thrown at kernel/util/httprequest.go:351

	resp, err := ssrfSafeClient.Do(req)
	if err != nil {
		return 0, "", "", errors.New("request failed: " + err.Error())
	}
	if resp == nil {
		return 0, "", "", errors.New("nil response")
	}
	defer resp.Body.Close()

	statusCode = resp.StatusCode
	contentType = resp.Header.Get("Content-Type")

	maxReadBytes := int64(maxHTTPRequestBytes)
	if !isTextContentType(contentType) {
		maxReadBytes = maxHTTPRequestFileBytes
	}
	// ContentLength 为 -1（chunked）时跳过大小预检，交由 LimitReader 兜底截断。
	if resp.ContentLength > maxReadBytes {
		return statusCode, contentType, "", errors.New("response too large")
	}

	respBody, rerr := io.ReadAll(io.LimitReader(resp.Body, maxReadBytes))
	if rerr != nil {
		return statusCode, contentType, "", errors.New("read body failed: " + rerr.Error())
	}

	// 二进制响应落盘，返回文件路径，供智能体按需进一步处理。
	if !isTextContentType(contentType) {
		importDir := filepath.Join(TempDir, "import")
		if merr := os.MkdirAll(importDir, 0755); merr != nil {
			return statusCode, contentType, "", errors.New("create import dir failed: " + merr.Error())
		}
		filename := extractFilename(rawURL, contentType)
		filePath := filepath.Join(importDir, filename)
		if werr := os.WriteFile(filePath, respBody, 0644); werr != nil {
			return statusCode, contentType, "", errors.New("write file failed: " + werr.Error())
		}

View on GitHub (pinned to 9f775e8a12)