a valid URL was expected (note: local items can download dat

Error message

a valid URL was expected (note: local items can download data too): %s

What it means

downloadDataSet validates each data source's SourceURL with url.Parse before downloading. This error is thrown when the parsed URL has no scheme (e.g. "example.com/file.txt" or a bare local path instead of "http(s)://..." or "file://..."). The hub requires an explicitly parseable absolute URL for data sources attached to a collection/parser.

Source

Thrown at pkg/hubops/download.go:126

		if err := dec.Decode(data); err != nil {
			if errors.Is(err, io.EOF) {
				break
			}

			return needReload, fmt.Errorf("while reading file: %w", err)
		}

		for _, dataS := range data.Data {
			if dataS.SourceURL == "" {
				continue
			}

			// twopenny validation
			if u, err := url.Parse(dataS.SourceURL); err != nil {
				return false, err
			} else if u.Scheme == "" {
				return false, fmt.Errorf("a valid URL was expected (note: local items can download data too): %s", dataS.SourceURL)
			}

			// XXX: check context cancellation
			destPath, err := cwhub.SafePath(dataFolder, dataS.DestPath)
			if err != nil {
				return needReload, err
			}

			d := downloader.
				New().
				WithHTTPClient(cwhub.HubClient).
				WithMakeDirs(true).
				ToFile(destPath).
				CompareContent().
				BeforeRequest(func(req *http.Request) {
					fmt.Fprintf(os.Stdout, "downloading %s\n", req.URL)
				}).
				WithLogger(log.WithField("url", dataS.SourceURL))

View on GitHub (pinned to 909b515798)

Solutions

Open the item's data manifest (e.g. <hub>/data/<item>/<item>.data.yaml) and prefix the SourceURL with a valid scheme (https:// or file://).
If the data is local, keep the file next to the item and use a proper file:// URL, since local items can still declare downloadable data.
Update the item from the hub (`cscli hub update && cscli hub upgrade`) if the manifest came from upstream and was fixed there.
Verify the fix with `cscli hub list -a` / re-run the download command.

Example fix

// before (data manifest)
source: blocklists/mylist.txt
// after
source: https://raw.example.org/blocklists/mylist.txt

Defensive patterns

Strategy: validation

Validate before calling

u, err := url.Parse(dataS.SourceURL)
if err != nil || u.Scheme == "" {
    return fmt.Errorf("data source %q is not an absolute URL", dataS.SourceURL)
}

Prevention

Always write data source URLs with an explicit scheme (https:// or file://) in .data.yaml manifests.
Validate item manifests with a lint step before shipping or installing them.
Prefer installing items from the official hub rather than hand-editing manifests.

When it happens

Trigger: Occurs during `crowdsec hub update`/hub operations that download data files (DownloadDataIfNeeded, download Run) when a data source entry in an item's .data.yaml (dataS.SourceURL) is missing its scheme — e.g. written as `source: mydata.txt` or `source: ./data/x.txt`.

Common situations: Hand-editing a data manifest and omitting `http://`/`https://`; copying a relative file path into SourceURL; a hub item shipped with a malformed data section; typos like `htp://` producing an empty/unrecognized scheme.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/82f376909d7f5bbe. Report an issue: GitHub.