crowdsecurity/crowdsec · error

while getting data: %w

Error message

while getting data: %w

What it means

Generic wrapping of any failure returned by d.Download(ctx, dataS.SourceURL) while fetching a data file declared by a hub item. The wrapper adds context ("while getting data") while preserving the underlying cause via %w, so the root reason (network, HTTP status, filesystem) is in the wrapped error.

Source

Thrown at pkg/hubops/download.go:153

			d := downloader.
				New().
				WithHTTPClient(cwhub.HubClient).
				WithMakeDirs(true).
				ToFile(destPath).
				CompareContent().
				BeforeRequest(func(req *http.Request) {
					fmt.Fprintf(os.Stdout, "downloading %s\n", req.URL)
				}).
				WithLogger(log.WithField("url", dataS.SourceURL))

			if !force {
				d = d.WithLastModified().
					WithShelfLife(7 * 24 * time.Hour)
			}

			downloaded, err := d.Download(ctx, dataS.SourceURL)
			if err != nil {
				return needReload, fmt.Errorf("while getting data: %w", err)
			}

			needReload = needReload || downloaded
		}
	}

	return needReload, nil
}

func (c *DownloadCommand) Run(ctx context.Context, plan *ActionPlan) error {
	i := c.Item

	fmt.Fprintf(os.Stdout, "downloading %s\n", colorizeItemName(i.FQName()))

	// ensure that target file is within target dir
	finalPath, err := i.PathForDownload()
	if err != nil {
		return err

View on GitHub (pinned to 909b515798)

Solutions

1. Read the wrapped cause in the log/output and fix that root issue first (network, proxy, URL, permissions).
2. Test the SourceURL with `curl -I <url>` to confirm reachability.
3. Configure HTTP proxy environment variables (HTTP_PROXY/HTTPS_PROXY) if behind a proxy.
4. Retry later if the upstream is temporarily down; for persistent 4xx/5xx, update the hub item to a fixed data URL.

resp, err := http.Head(dataS.SourceURL)
if err != nil || resp.StatusCode != http.StatusOK {
    return fmt.Errorf("data URL %s unreachable", dataS.SourceURL)
}

if err := downloadData(); err != nil {
    var netErr net.Error
    if errors.As(err, &netErr) {
        // backoff and retry later
    }
    log.Fatalf("while getting data: %v", err)
}

When it happens

Trigger: Any hub operation calling downloadDataSet (DownloadDataIfNeeded or download Run) where the HTTP GET of a data source fails: unreachable host, DNS failure, non-200 response, TLS error, timeout, or disk write failure in the data folder.

Common situations: Downloading blocklists behind a corporate proxy; the upstream data URL is dead or rate-limiting; no internet access on an air-gapped host; permission problems in /var/lib/crowdsec/data.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• while downloading data for %s: %w
• %s: %w
• while downloading data for %s: %w
• unable to download data for parser '%s': %w
• failed to parse URL: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/754e1f923e05661d. Report an issue: GitHub.