{"record":{"id":"754e1f923e05661d","repo":"crowdsecurity/crowdsec","slug":"while-getting-data-w","errorCode":null,"errorMessage":"while getting data: %w","messagePattern":"while getting data: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/hubops/download.go","lineNumber":153,"sourceCode":"\t\t\td := downloader.\n\t\t\t\tNew().\n\t\t\t\tWithHTTPClient(cwhub.HubClient).\n\t\t\t\tWithMakeDirs(true).\n\t\t\t\tToFile(destPath).\n\t\t\t\tCompareContent().\n\t\t\t\tBeforeRequest(func(req *http.Request) {\n\t\t\t\t\tfmt.Fprintf(os.Stdout, \"downloading %s\\n\", req.URL)\n\t\t\t\t}).\n\t\t\t\tWithLogger(log.WithField(\"url\", dataS.SourceURL))\n\n\t\t\tif !force {\n\t\t\t\td = d.WithLastModified().\n\t\t\t\t\tWithShelfLife(7 * 24 * time.Hour)\n\t\t\t}\n\n\t\t\tdownloaded, err := d.Download(ctx, dataS.SourceURL)\n\t\t\tif err != nil {\n\t\t\t\treturn needReload, fmt.Errorf(\"while getting data: %w\", err)\n\t\t\t}\n\n\t\t\tneedReload = needReload || downloaded\n\t\t}\n\t}\n\n\treturn needReload, nil\n}\n\nfunc (c *DownloadCommand) Run(ctx context.Context, plan *ActionPlan) error {\n\ti := c.Item\n\n\tfmt.Fprintf(os.Stdout, \"downloading %s\\n\", colorizeItemName(i.FQName()))\n\n\t// ensure that target file is within target dir\n\tfinalPath, err := i.PathForDownload()\n\tif err != nil {\n\t\treturn err","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/hubops/download.go#L135-L171","documentation":"Generic wrapping of any failure returned by d.Download(ctx, dataS.SourceURL) while fetching a data file declared by a hub item. The wrapper adds context (\"while getting data\") while preserving the underlying cause via %w, so the root reason (network, HTTP status, filesystem) is in the wrapped error.","triggerScenarios":"Any hub operation calling downloadDataSet (DownloadDataIfNeeded or download Run) where the HTTP GET of a data source fails: unreachable host, DNS failure, non-200 response, TLS error, timeout, or disk write failure in the data folder.","commonSituations":"Downloading blocklists behind a corporate proxy; the upstream data URL is dead or rate-limiting; no internet access on an air-gapped host; permission problems in /var/lib/crowdsec/data.","solutions":["Read the wrapped cause in the log/output and fix that root issue first (network, proxy, URL, permissions).","Test the SourceURL with `curl -I <url>` to confirm reachability.","Configure HTTP proxy environment variables (HTTP_PROXY/HTTPS_PROXY) if behind a proxy.","Retry later if the upstream is temporarily down; for persistent 4xx/5xx, update the hub item to a fixed data URL."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"resp, err := http.Head(dataS.SourceURL)\nif err != nil || resp.StatusCode != http.StatusOK {\n    return fmt.Errorf(\"data URL %s unreachable\", dataS.SourceURL)\n}","typeGuard":null,"tryCatchPattern":"if err := downloadData(); err != nil {\n    var netErr net.Error\n    if errors.As(err, &netErr) {\n        // backoff and retry later\n    }\n    log.Fatalf(\"while getting data: %v\", err)\n}","preventionTips":["Test each data SourceURL with curl -I before relying on it.","Set HTTP_PROXY/HTTPS_PROXY when operating behind a proxy.","Schedule periodic hub updates so dead upstream data URLs are replaced promptly.","Monitor crowdsec logs for repeated download failures."],"tags":["network","download","hub","wrapped-error"],"backgroundTag":"http-request-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}