What causes "response too large" in siyuan-note/siyuan?

Fetch a smaller URL — WebFetch is meant for pages/text, not large downloads.

response too large

Error message

response too large

What it means

Returned at webfetch.go:70 when resp.ContentLength exceeds the cap declared in webfetch.go:36-39: 5 MiB for text/html or text/plain, 10 MiB for any other content type. Note the check uses the server-declared Content-Length header, so chunked/streaming responses (ContentLength == -1) skip this guard and are instead capped by io.LimitReader at read time.

Source

Thrown at kernel/util/webfetch.go:70

	}

	resp, err := httpclient.NewBrowserRequest().Get(rawURL)
	if err != nil {
		return "", errors.New("fetch failed: " + err.Error())
	}
	defer resp.Body.Close()

	if resp.StatusCode >= 400 {
		return "", fmt.Errorf("HTTP %d", resp.StatusCode)
	}

	contentType := resp.Header.Get("Content-Type")
	maxReadBytes := int64(maxWebFetchBytes)
	if !strings.HasPrefix(contentType, "text/html") && !strings.HasPrefix(contentType, "text/plain") {
		maxReadBytes = maxWebFetchFileBytes
	}
	if resp.ContentLength > maxReadBytes {
		return "", errors.New("response too large")
	}

	body, err := io.ReadAll(io.LimitReader(resp.Body, maxReadBytes))
	if err != nil {
		return "", errors.New("read body failed: " + err.Error())
	}

	if !strings.HasPrefix(contentType, "text/html") && !strings.HasPrefix(contentType, "text/plain") {
		importDir := filepath.Join(TempDir, "import")
		if merr := os.MkdirAll(importDir, 0755); merr != nil {
			return "", errors.New("create import dir failed: " + merr.Error())
		}
		filename := extractFilename(rawURL, contentType)
		filePath := filepath.Join(importDir, filename)
		if werr := os.WriteFile(filePath, body, 0644); werr != nil {
			return "", errors.New("write file failed: " + werr.Error())
		}
		return fmt.Sprintf("Saved to: %s (%d bytes)", filePath, len(body)), nil

View on GitHub (pinned to 251596fc0d)

Solutions

Fetch a smaller URL — WebFetch is meant for pages/text, not large downloads.
Download the large file out-of-band (browser, wget) and import it separately.
If you genuinely need it, raise maxWebFetchFileBytes in kernel/util/webfetch.go and rebuild the kernel (local change only).

Defensive patterns

Strategy: validation

Validate before calling

// HEAD the URL to learn the declared size before paying for the GET.
func declaredTooLarge(rawURL string) (bool, error) {
    resp, err := httpclient.NewBrowserRequest().Head(rawURL)
    if err != nil {
        return false, err
    }
    defer resp.Body.Close()
    return resp.ContentLength > int64(10*1024*1024), nil
}

Type guard

func isResponseTooLarge(err error) bool {
    return err != nil && err.Error() == "response too large"
}

Try / catch

out, err := util.WebFetch(raw, "markdown")
if err != nil && isResponseTooLarge(err) {
    // surface a user action: pick a smaller URL or download out-of-band
    return errors.New("target is too large to fetch via WebFetch; download it directly")
}

Prevention

Reserve WebFetch for HTML/text pages, not large assets.
HEAD-check Content-Length for untrusted URLs before fetching.
Remember chunked responses (no Content-Length) bypass this guard and are capped at read time instead.

When it happens

Trigger: Pointing WebFetch at a large PDF/image/archive whose declared Content-Length exceeds 10 MiB, or an HTML/plain page larger than 5 MiB.

Common situations: User pastes a direct media-asset URL expecting page text; fetching a large dataset dump; CDN reporting true Content-Length on a big file.

Related errors

AI-assisted analysis of siyuan-note/siyuan@251596fc0d (2026-08-12). Data as JSON: /api/errors/cfbb11a390cec043. Report an issue: GitHub.