{"record":{"id":"76b401b04548476f","repo":"crowdsecurity/crowdsec","slug":"while-downloading-data-for-s-w","errorCode":null,"errorMessage":"while downloading data for %s: %w","messagePattern":"while downloading data for (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/hubops/datarefresh.go","lineNumber":23,"sourceCode":"\t\"fmt\"\n\t\"os\"\n\n\t\"github.com/crowdsecurity/crowdsec/pkg/cwhub\"\n)\n\n// XXX: TODO: temporary for hubtests, but will have to go.\n// DownloadDataIfNeeded downloads the data set for the item.\nfunc DownloadDataIfNeeded(ctx context.Context, hub *cwhub.Hub, item *cwhub.Item, force bool) (bool, error) {\n\titemFile, err := os.Open(item.State.LocalPath)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"while opening %s: %w\", item.State.LocalPath, err)\n\t}\n\n\tdefer itemFile.Close()\n\n\tneedReload, err := downloadDataSet(ctx, hub.GetDataDir(), force, itemFile)\n\tif err != nil {\n\t\treturn needReload, fmt.Errorf(\"while downloading data for %s: %w\", item.State.LocalPath, err)\n\t}\n\n\treturn needReload, nil\n}\n\n// DataRefreshCommand updates the data files associated with the installed hub items.\ntype DataRefreshCommand struct {\n\tForce bool\n}\n\nfunc NewDataRefreshCommand(force bool) *DataRefreshCommand {\n\treturn &DataRefreshCommand{Force: force}\n}\n\nfunc (*DataRefreshCommand) Prepare(_ *ActionPlan) (bool, error) {\n\t// we can't prepare much at this point because we don't know which data files yet,\n\t// and items needs to be downloaded/updated\n\t// evertyhing will be done in Run()","sourceCodeStart":5,"sourceCodeEnd":41,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/hubops/datarefresh.go#L5-L41","documentation":"DownloadDataIfNeeded calls downloadDataSet to fetch and decode the remote data associated with a hub item. Any failure inside that download/decode pipeline is wrapped as 'while downloading data for %s: %w', identifying the item by its local path.","triggerScenarios":"Calling DownloadDataIfNeeded when the upstream data download fails: bad URL, network outage, HTTP error, or malformed/undecodable data content.","commonSituations":"Upstream data source (e.g. blocklist provider) is down or moved its URL; network egress blocked in the deployment; the item's data file format changed and decoding fails.","solutions":["Run 'cscli hub update' to refresh item metadata and data URLs","Check network connectivity to the data source URL","Inspect the wrapped inner error for the root cause (HTTP status vs decode error)","Reinstall the item to reset its data source configuration"],"exampleFix":"// before\nneedReload, err := hubops.DownloadDataIfNeeded(ctx, hub, item, force)\nif err != nil { log.Fatal(err) }\n// after\nneedReload, err := hubops.DownloadDataIfNeeded(ctx, hub, item, force)\nif err != nil {\n    log.Warnf(\"data refresh failed for %s: %v; will retry\", item.FQName(), err)\n}","handlingStrategy":"retry","validationCode":"resp, err := http.Head(dataURL)\nif err != nil || resp.StatusCode != http.StatusOK { return fmt.Errorf(\"data source unavailable: %s\", dataURL) }","typeGuard":null,"tryCatchPattern":"needReload, err := hubops.DownloadDataIfNeeded(ctx, hub, item, force)\nif err != nil {\n    return retry.WithBackoff(ctx, 3, func() error {\n        _, err := hubops.DownloadDataIfNeeded(ctx, hub, item, force)\n        return err\n    })\n}","preventionTips":["Check network egress before hub operations","Refresh hub metadata regularly so data URLs stay valid","Wrap with retries and timeouts against transient network errors"],"tags":["network","hub","download"],"backgroundTag":"http-request-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}