crowdsecurity/crowdsec · error

failed to parse URL: %w

Error message

failed to parse URL: %w

What it means

After substituting the branch and remote path into the downloader's URL template, urlTo parses the result with url.Parse. If the resulting string is not a parseable URL (e.g. invalid characters, bad scheme), it returns 'failed to parse URL' wrapping the net/url error, failing FetchIndex/FetchContent.

Source

Thrown at pkg/cwhub/download.go:46

}

// ContentProvider retrieves and writes the YAML files with the item content.
type ContentProvider interface {
	FetchContent(ctx context.Context, remotePath, destPath, wantHash string, logger *logrus.Logger) (bool, string, error)
}

// urlTo builds the URL to download a file from the remote hub.
func (d *Downloader) urlTo(remotePath string) (*url.URL, error) {
	// the template must contain two string placeholders
	if fmt.Sprintf(d.URLTemplate, "%s", "%s") != d.URLTemplate {
		return nil, fmt.Errorf("invalid URL template '%s'", d.URLTemplate)
	}

	raw := fmt.Sprintf(d.URLTemplate, d.Branch, remotePath)

	parsed, err := url.Parse(raw)
	if err != nil {
		return nil, fmt.Errorf("failed to parse URL: %w", err)
	}

	return parsed, nil
}

// FetchIndex downloads the index from the hub and writes it to the filesystem.
// It uses a temporary file to avoid partial downloads, and won't overwrite the original
// if it has not changed.
// Return true if the file has been updated, false if already up to date.
func (d *Downloader) FetchIndex(ctx context.Context, destPath string, withContent bool, logger *logrus.Logger) (downloaded bool, err error) {
	url, err := d.urlTo(".index.json")
	if err != nil {
		return false, fmt.Errorf("failed to build hub index request: %w", err)
	}

	if withContent {
		q := url.Query()
		q.Set("with_content", "true")

View on GitHub (pinned to 909b515798)

Solutions

1. Inspect the wrapped net/url error for the offending character/position.
2. Remove stray spaces/quotes/control characters from the hub URL in config.yaml.
3. Validate the final URL by pasting it into a browser or `curl -I`.
4. If the offending character comes from the remote path (index file name), update/re-fetch the hub index; this may be an upstream index issue.

Example fix

// before (config.yaml)
hub_url: " https://hub.example.com/%s/%s"  # leading space
// after
hub_url: "https://hub.example.com/%s/%s"

u, err := url.Parse(fmt.Sprintf(template, branch, remotePath))
if err != nil {
    return fmt.Errorf("resulting hub URL is invalid: %w", err)
}

u, err := d.urlTo(remotePath)
if err != nil {
    return fmt.Errorf("cannot build hub URL: %w", err)
}

When it happens

Trigger: URLTemplate (after branch+remotePath substitution) yields a string url.Parse rejects: control characters or spaces in the hub URL or remote path, an empty/malformed scheme, or a path containing invalid percent sequences.

Common situations: Whitespace or quotes accidentally included in the hub_url config value; a remote file path with characters that break URL syntax; a misconfigured custom mirror URL with typos.

Understand the failure class

Background: "Invalid URL" / "URL cannot be empty": fix the malformed or missing URL behind request-construction failures — this error's family across 50 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to build hub index request: %w
• failed to build request: %w
• invalid URL template '%s'
• while downloading data for %s: %w
• while getting data: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/a79a1115e885cc33. Report an issue: GitHub.