siyuan-note/siyuan · error

response too large

Error message

response too large

What it means

WebFetch enforces a size cap before reading the body: if resp.ContentLength exceeds maxReadBytes (maxWebFetchBytes for HTML/text content types, the smaller maxWebFetchFileBytes otherwise), it aborts with 'response too large'. The limit is declared via Content-Length, so chunked responses without a length are only partially read instead of rejected here.

Solutions

1. Fetch a smaller resource or a more specific URL (direct page, not a file dump)
2. Check the resource's Content-Length with a HEAD request before calling WebFetch
3. If the file is legitimately needed, download it through a different path that streams to disk instead of WebFetch
4. Increase the maxWebFetchBytes/maxWebFetchFileBytes limits only if memory budgets allow

Example fix

// before
_, err := WebFetch("https://example.com/big-dump.zip", "markdown")

// after
resp, _ := http.Head("https://example.com/big-dump.zip")
if resp.ContentLength <= 5*1024*1024 {
    _, err = WebFetch("https://example.com/big-dump.zip", "markdown")
} else {
    err = errors.New("use streaming download instead")
}

resp, err := http.Head(url)
if err == nil && resp.ContentLength > maxWebFetchBytes {
    return errors.New("skipping: payload exceeds WebFetch limit; use streaming download")
}

When it happens

Trigger: Fetching a very large HTML page or a non-HTML file (binary, PDF, archive) whose Content-Length exceeds the configured byte limit.

Common situations: Pointing WebFetch at large downloadable files instead of pages; fetching huge generated HTML dashboards; a site serving a multi-hundred-MB payload where a small file was expected.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• response too large
• custom emoji file is too large
• Agent capability name and description are required
• asset path [ ] does not match data path [ ]
• attribute view custom colors must not be empty

AI-assisted analysis of siyuan-note/siyuan@9f775e8a12 (2026-09-19). Data as JSON: /api/errors/cfbb11a390cec043. Report an issue: GitHub.

Appendix: source

Thrown at kernel/util/webfetch.go:69

	}

	resp, err := ssrfSafeClient.Get(rawURL)
	if err != nil {
		return "", errors.New("fetch failed: " + err.Error())
	}
	defer resp.Body.Close()

	if resp.StatusCode >= 400 {
		return "", fmt.Errorf("HTTP %d", resp.StatusCode)
	}

	contentType := resp.Header.Get("Content-Type")
	maxReadBytes := int64(maxWebFetchBytes)
	if !strings.HasPrefix(contentType, "text/html") && !strings.HasPrefix(contentType, "text/plain") {
		maxReadBytes = maxWebFetchFileBytes
	}
	if resp.ContentLength > maxReadBytes {
		return "", errors.New("response too large")
	}

	body, err := io.ReadAll(io.LimitReader(resp.Body, maxReadBytes))
	if err != nil {
		return "", errors.New("read body failed: " + err.Error())
	}

	if !strings.HasPrefix(contentType, "text/html") && !strings.HasPrefix(contentType, "text/plain") {
		importDir := filepath.Join(TempDir, "import")
		if merr := os.MkdirAll(importDir, 0755); merr != nil {
			return "", errors.New("create import dir failed: " + merr.Error())
		}
		filename := extractFilename(rawURL, contentType)
		filePath := filepath.Join(importDir, filename)
		if werr := os.WriteFile(filePath, body, 0644); werr != nil {
			return "", errors.New("write file failed: " + werr.Error())
		}
		return fmt.Sprintf("Saved to: %s (%d bytes)", filePath, len(body)), nil

View on GitHub (pinned to 9f775e8a12)