crowdsecurity/crowdsec · error

unable to download data for parser '%s': %w

Error message

unable to download data for parser '%s': %w

What it means

InstallHub installs parser data files into the test runtime via hubops.DownloadDataIfNeeded for each installed parser; any download failure is wrapped as 'unable to download data for parser <name>: %w'. The hub items are installed but their associated data files (e.g. geoip databases) could not be fetched.

Source

Thrown at pkg/hubtest/hubtest_item.go:247

	// load installed hub
	hub, err := cwhub.NewHub(t.RuntimeHubConfig, nil)
	if err != nil {
		return err
	}

	if err := hub.Load(); err != nil {
		return err
	}

	// prevent concurrent downloads of the same file
	downloadMutex.Lock()
	defer downloadMutex.Unlock()

	// install data for parsers if needed
	for _, item := range hub.GetInstalledByType(cwhub.PARSERS, true) {
		if _, err := hubops.DownloadDataIfNeeded(ctx, hub, item, false); err != nil {
			return fmt.Errorf("unable to download data for parser '%s': %w", item.Name, err)
		}

		log.Debugf("parser '%s' installed successfully in runtime environment", item.Name)
	}

	// install data for scenarios if needed
	for _, item := range hub.GetInstalledByType(cwhub.SCENARIOS, true) {
		if _, err := hubops.DownloadDataIfNeeded(ctx, hub, item, false); err != nil {
			return fmt.Errorf("unable to download data for parser '%s': %w", item.Name, err)
		}

		log.Debugf("scenario '%s' installed successfully in runtime environment", item.Name)
	}

	// install data for postoverflows if needed
	for _, item := range hub.GetInstalledByType(cwhub.POSTOVERFLOWS, true) {
		if _, err := hubops.DownloadDataIfNeeded(ctx, hub, item, false); err != nil {
			return fmt.Errorf("unable to download data for parser '%s': %w", item.Name, err)

View on GitHub (pinned to 909b515798)

Solutions

1. Read the wrapped error: DNS/timeout/refused -> fix network or proxy settings (set HTTPS_PROXY); 404 -> update the hub index.
2. Run `cscli hub update` / refresh .index.json so data URLs are current.
3. Pre-download or vendor the data files into the runtime directory so DownloadDataIfNeeded finds them locally and skips the download.
4. Re-run with network access enabled in CI (e.g. allow-list hub data hosts, or use a caching proxy).

Example fix

null

resp, err := http.Head(dataURL)
if err != nil || resp.StatusCode != 200 {
    log.Printf("data URL unreachable, check network/proxy before running hub tests")
}

err := item.Run(ctx)
if err != nil && strings.Contains(err.Error(), "unable to download data") {
    // transient? retry with backoff; else fix network/proxy and re-run
    time.Sleep(2 * time.Second)
    err = item.Run(ctx)
}

When it happens

Trigger: HubTestItem.Run -> InstallHub with no network access to the data URL, an invalid/expired data download URL in the hub index, or hub index/data URL misconfiguration.

Common situations: CI without internet access; corporate proxy/firewall blocking hub data hosts; stale .index.json referencing moved data files; TLS/certificate problems; rate limiting on the data CDN.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• while downloading data for %s: %w
• while getting data: %w
• failed to parse URL: %w
• failed to build hub index request: %w
• failed to build request: %w

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/4daa9391e731cd33. Report an issue: GitHub.