{"record":{"id":"2b7e811c373b94a1","repo":"siyuan-note/siyuan","slug":"url-must-start-with-http-or-https-2b7e81","errorCode":null,"errorMessage":"URL must start with http:// or https://","messagePattern":"URL must start with http:// or https://","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"kernel/util/webfetch.go","lineNumber":44,"sourceCode":"\t\"path\"\n\t\"path/filepath\"\n\t\"strings\"\n\n\t\"github.com/88250/gulu\"\n\t\"github.com/88250/lute\"\n\t\"github.com/siyuan-note/httpclient\"\n)\n\nconst (\n\tmaxWebFetchBytes     = 5 * 1024 * 1024  // text/html, text/plain\n\tmaxWebFetchFileBytes = 10 * 1024 * 1024 // file/image download\n\tmaxWebFetchChars     = 50000\n)\n\nfunc WebFetch(rawURL, format string) (string, error) {\n\tu, err := url.Parse(rawURL)\n\tif err != nil || (u.Scheme != \"http\" && u.Scheme != \"https\") {\n\t\treturn \"\", errors.New(\"URL must start with http:// or https://\")\n\t}\n\tif u.Host == \"\" {\n\t\treturn \"\", errors.New(\"URL has no host\")\n\t}\n\n\tif err := CheckHostSSRF(u.Hostname()); err != nil {\n\t\treturn \"\", err\n\t}\n\n\tresp, err := httpclient.NewBrowserRequest().Get(rawURL)\n\tif err != nil {\n\t\treturn \"\", errors.New(\"fetch failed: \" + err.Error())\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode >= 400 {\n\t\treturn \"\", fmt.Errorf(\"HTTP %d\", resp.StatusCode)\n\t}","sourceCodeStart":26,"sourceCodeEnd":62,"githubUrl":"https://github.com/siyuan-note/siyuan/blob/251596fc0de2f9528c00c224252fd073a99973f4/kernel/util/webfetch.go#L26-L62","documentation":"Thrown by WebFetch at the top when url.Parse fails or the scheme is neither http nor https. It is the scheme precondition before any network activity. Only http(s) URLs are accepted; ftp/file/javascript/data schemes are rejected.","triggerScenarios":"Passing a URL without a scheme (e.g. 'example.com/page'); a URL with a non-http scheme (file://, ftp://); a malformed URL that url.Parse rejects; a relative path.","commonSituations":"User submitted a bare domain; the URL field was auto-filled without https://; an internal caller passed a path rather than a full URL; copy-paste dropped the scheme.","solutions":["Prepend 'https://' to bare-domain inputs before calling WebFetch.","Validate the scheme client-side and reject non-http(s) values.","Use url.Parse upstream and inspect u.Scheme to give a precise error."],"exampleFix":"// before\nutil.WebFetch(\"example.com/page\", \"markdown\")\n\n// after\nutil.WebFetch(\"https://example.com/page\", \"markdown\")","handlingStrategy":"validation","validationCode":"u, err := url.Parse(rawURL)\nif err != nil || (u.Scheme != \"http\" && u.Scheme != \"https\") {\n    if !strings.Contains(rawURL, \"://\") {\n        rawURL = \"https://\" + rawURL // auto-fix bare domains\n    }\n    u, err = url.Parse(rawURL)\n    if err != nil || (u.Scheme != \"http\" && u.Scheme != \"https\") {\n        return errors.New(\"only http and https URLs are supported\")\n    }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always include the https:// scheme when collecting URLs from users.","Reject non-http(s) schemes at the input boundary.","Auto-prepend https:// to bare-domain inputs in the UI."],"tags":["webfetch","validation","url"],"backgroundTag":null,"analyzedSha":"251596fc0de2f9528c00c224252fd073a99973f4","analyzedAt":"2026-08-12T21:18:37.123Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}