{"record":{"id":"82f376909d7f5bbe","repo":"crowdsecurity/crowdsec","slug":"a-valid-url-was-expected-note-local-items-can-do","errorCode":null,"errorMessage":"a valid URL was expected (note: local items can download data too): %s","messagePattern":"a valid URL was expected \\(note: local items can download data too\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/hubops/download.go","lineNumber":126,"sourceCode":"\n\t\tif err := dec.Decode(data); err != nil {\n\t\t\tif errors.Is(err, io.EOF) {\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\treturn needReload, fmt.Errorf(\"while reading file: %w\", err)\n\t\t}\n\n\t\tfor _, dataS := range data.Data {\n\t\t\tif dataS.SourceURL == \"\" {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\t// twopenny validation\n\t\t\tif u, err := url.Parse(dataS.SourceURL); err != nil {\n\t\t\t\treturn false, err\n\t\t\t} else if u.Scheme == \"\" {\n\t\t\t\treturn false, fmt.Errorf(\"a valid URL was expected (note: local items can download data too): %s\", dataS.SourceURL)\n\t\t\t}\n\n\t\t\t// XXX: check context cancellation\n\t\t\tdestPath, err := cwhub.SafePath(dataFolder, dataS.DestPath)\n\t\t\tif err != nil {\n\t\t\t\treturn needReload, err\n\t\t\t}\n\n\t\t\td := downloader.\n\t\t\t\tNew().\n\t\t\t\tWithHTTPClient(cwhub.HubClient).\n\t\t\t\tWithMakeDirs(true).\n\t\t\t\tToFile(destPath).\n\t\t\t\tCompareContent().\n\t\t\t\tBeforeRequest(func(req *http.Request) {\n\t\t\t\t\tfmt.Fprintf(os.Stdout, \"downloading %s\\n\", req.URL)\n\t\t\t\t}).\n\t\t\t\tWithLogger(log.WithField(\"url\", dataS.SourceURL))","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/hubops/download.go#L108-L144","documentation":"downloadDataSet validates each data source's SourceURL with url.Parse before downloading. This error is thrown when the parsed URL has no scheme (e.g. \"example.com/file.txt\" or a bare local path instead of \"http(s)://...\" or \"file://...\"). The hub requires an explicitly parseable absolute URL for data sources attached to a collection/parser.","triggerScenarios":"Occurs during `crowdsec hub update`/hub operations that download data files (DownloadDataIfNeeded, download Run) when a data source entry in an item's .data.yaml (dataS.SourceURL) is missing its scheme — e.g. written as `source: mydata.txt` or `source: ./data/x.txt`.","commonSituations":"Hand-editing a data manifest and omitting `http://`/`https://`; copying a relative file path into SourceURL; a hub item shipped with a malformed data section; typos like `htp://` producing an empty/unrecognized scheme.","solutions":["Open the item's data manifest (e.g. <hub>/data/<item>/<item>.data.yaml) and prefix the SourceURL with a valid scheme (https:// or file://).","If the data is local, keep the file next to the item and use a proper file:// URL, since local items can still declare downloadable data.","Update the item from the hub (`cscli hub update && cscli hub upgrade`) if the manifest came from upstream and was fixed there.","Verify the fix with `cscli hub list -a` / re-run the download command."],"exampleFix":"// before (data manifest)\nsource: blocklists/mylist.txt\n// after\nsource: https://raw.example.org/blocklists/mylist.txt","handlingStrategy":"validation","validationCode":"u, err := url.Parse(dataS.SourceURL)\nif err != nil || u.Scheme == \"\" {\n    return fmt.Errorf(\"data source %q is not an absolute URL\", dataS.SourceURL)\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always write data source URLs with an explicit scheme (https:// or file://) in .data.yaml manifests.","Validate item manifests with a lint step before shipping or installing them.","Prefer installing items from the official hub rather than hand-editing manifests."],"tags":["url-validation","hub","data-download","config"],"backgroundTag":"invalid-url-format","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}