{"record":{"id":"58f5557dffad66e9","repo":"Tencent/WeKnora","slug":"url-rejected-w","errorCode":null,"errorMessage":"URL rejected: %w","messagePattern":"URL rejected: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/datasource/connector/rss/client.go","lineNumber":50,"sourceCode":"\thttpClient *http.Client\n\theaders    map[string]string\n}\n\nfunc newClient(headers map[string]string) *client {\n\tcfg := utils.DefaultSSRFSafeHTTPClientConfig()\n\tcfg.Timeout = requestTimeout\n\treturn &client{\n\t\thttpClient: utils.NewSSRFSafeHTTPClient(cfg),\n\t\theaders:    headers,\n\t}\n}\n\n// fetch retrieves rawURL with SSRF validation and size limiting. Custom auth\n// headers are only attached when withAuthHeaders is true (feed fetches); article\n// pages on third-party domains must not receive feed credentials.\nfunc (c *client) fetch(ctx context.Context, rawURL string, maxSize int64, withAuthHeaders bool) ([]byte, error) {\n\tif err := utils.ValidateURLForSSRF(rawURL); err != nil {\n\t\treturn nil, fmt.Errorf(\"URL rejected: %w\", err)\n\t}\n\tif _, err := url.Parse(rawURL); err != nil {\n\t\treturn nil, fmt.Errorf(\"invalid URL: %w\", err)\n\t}\n\n\tctx, cancel := context.WithTimeout(ctx, requestTimeout)\n\tdefer cancel()\n\n\treq, err := http.NewRequestWithContext(ctx, http.MethodGet, rawURL, nil)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif withAuthHeaders {\n\t\tfor k, v := range c.headers {\n\t\t\treq.Header.Set(k, v)\n\t\t}\n\t}","sourceCodeStart":32,"sourceCodeEnd":68,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/datasource/connector/rss/client.go#L32-L68","documentation":"The RSS connector's fetch validates every URL with utils.ValidateURLForSSRF before issuing a request, defending against server-side request forgery (requests to internal/loopback/link-local addresses). Rejected URLs produce \"URL rejected: %w\" with the validator's reason.","triggerScenarios":"fetchFeed or extractArticle passes a feed/article URL that resolves to or specifies a blocked target: localhost, 127.0.0.0/8, 10.x/172.16.x/192.168.x private ranges, 169.254.x link-local (incl. cloud metadata endpoints), non-HTTP schemes, or a hostname resolving to those IPs.","commonSituations":"Feed points at an internal dev URL (http://localhost:8080/feed.xml), self-hosted RSS behind a private network, hostnames resolving to private IPs in container/K8s clusters, metadata endpoints like http://169.254.169.254.","solutions":["Use a publicly resolvable HTTPS URL for the feed","If the feed is internal by design, deploy with SSRF validation allowlisting or run the fetch from an allowed network component (per your deployment's policy)","Verify the hostname's DNS resolves to a public IP (dig/nslookup)","Ensure the URL scheme is http/https on a standard port"],"exampleFix":"// before\nfetch(ctx, \"http://localhost:8080/rss\")\n// after\nfetch(ctx, \"https://example.com/feed.xml\")","handlingStrategy":"validation","validationCode":"u, err := url.Parse(rawURL)\nif err != nil || (u.Scheme != \"http\" && u.Scheme != \"https\") { return errors.New(\"must be a public http(s) URL\") }\naddrs, err := net.LookupHost(u.Hostname())\nif err != nil { return err }\nfor _, a := range addrs { if ip := net.ParseIP(a); ip != nil && (ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast()) { return errors.New(\"private IP not allowed\") } }","typeGuard":"func isPublicHTTPURL(raw string) bool {\n    u, err := url.Parse(raw)\n    if err != nil || u.Hostname() == \"\" { return false }\n    ip := net.ParseIP(u.Hostname())\n    if ip != nil { return !(ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast()) }\n    return u.Scheme == \"http\" || u.Scheme == \"https\"\n}","tryCatchPattern":"body, err := c.fetch(ctx, feedURL, maxSize, true)\nif err != nil {\n    if strings.Contains(err.Error(), \"URL rejected\") {\n        return fmt.Errorf(\"feed URL not allowed (public URLs only): %s\", feedURL)\n    }\n    return err\n}","preventionTips":["Only subscribe to publicly reachable feeds","Pre-verify feed hostnames resolve to public IPs before configuring","Avoid internal hostnames (localhost, *.local, 169.254.x) in feed URLs","Understand the connector blocks private ranges by design to prevent SSRF"],"tags":["rss","ssrf","security","url-validation"],"backgroundTag":"ssrf-url-rejected","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}