Tencent/WeKnora · error

URL rejected: %w

Error message

URL rejected: %w

What it means

The RSS connector's fetch validates every URL with utils.ValidateURLForSSRF before issuing a request, defending against server-side request forgery (requests to internal/loopback/link-local addresses). Rejected URLs produce "URL rejected: %w" with the validator's reason.

Source

Thrown at internal/datasource/connector/rss/client.go:50

	httpClient *http.Client
	headers    map[string]string
}

func newClient(headers map[string]string) *client {
	cfg := utils.DefaultSSRFSafeHTTPClientConfig()
	cfg.Timeout = requestTimeout
	return &client{
		httpClient: utils.NewSSRFSafeHTTPClient(cfg),
		headers:    headers,
	}
}

// fetch retrieves rawURL with SSRF validation and size limiting. Custom auth
// headers are only attached when withAuthHeaders is true (feed fetches); article
// pages on third-party domains must not receive feed credentials.
func (c *client) fetch(ctx context.Context, rawURL string, maxSize int64, withAuthHeaders bool) ([]byte, error) {
	if err := utils.ValidateURLForSSRF(rawURL); err != nil {
		return nil, fmt.Errorf("URL rejected: %w", err)
	}
	if _, err := url.Parse(rawURL); err != nil {
		return nil, fmt.Errorf("invalid URL: %w", err)
	}

	ctx, cancel := context.WithTimeout(ctx, requestTimeout)
	defer cancel()

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, rawURL, nil)
	if err != nil {
		return nil, err
	}

	if withAuthHeaders {
		for k, v := range c.headers {
			req.Header.Set(k, v)
		}
	}

View on GitHub (pinned to 988cbb0330)

Solutions

1. Use a publicly resolvable HTTPS URL for the feed
2. If the feed is internal by design, deploy with SSRF validation allowlisting or run the fetch from an allowed network component (per your deployment's policy)
3. Verify the hostname's DNS resolves to a public IP (dig/nslookup)
4. Ensure the URL scheme is http/https on a standard port

Example fix

// before
fetch(ctx, "http://localhost:8080/rss")
// after
fetch(ctx, "https://example.com/feed.xml")

u, err := url.Parse(rawURL)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") { return errors.New("must be a public http(s) URL") }
addrs, err := net.LookupHost(u.Hostname())
if err != nil { return err }
for _, a := range addrs { if ip := net.ParseIP(a); ip != nil && (ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast()) { return errors.New("private IP not allowed") } }

func isPublicHTTPURL(raw string) bool {
    u, err := url.Parse(raw)
    if err != nil || u.Hostname() == "" { return false }
    ip := net.ParseIP(u.Hostname())
    if ip != nil { return !(ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast()) }
    return u.Scheme == "http" || u.Scheme == "https"
}

body, err := c.fetch(ctx, feedURL, maxSize, true)
if err != nil {
    if strings.Contains(err.Error(), "URL rejected") {
        return fmt.Errorf("feed URL not allowed (public URLs only): %s", feedURL)
    }
    return err
}

When it happens

Trigger: fetchFeed or extractArticle passes a feed/article URL that resolves to or specifies a blocked target: localhost, 127.0.0.0/8, 10.x/172.16.x/192.168.x private ranges, 169.254.x link-local (incl. cloud metadata endpoints), non-HTTP schemes, or a hostname resolving to those IPs.

Common situations: Feed points at an internal dev URL (http://localhost:8080/feed.xml), self-hosted RSS behind a private network, hostnames resolving to private IPs in container/K8s clusters, metadata endpoints like http://169.254.169.254.

Related errors

• base_url SSRF validation failed: %w
• URL rejected for security reasons: %v
• docreader address failed SSRF validation: %w
• blocked by SSRF policy: %w
• zip URL blocked by SSRF check: %v

AI-assisted analysis of Tencent/WeKnora@988cbb0330 (2026-09-02). Data as JSON: /api/errors/58f5557dffad66e9. Report an issue: GitHub.