What causes "failed to parse index: %w" in crowdsecurity/crowdsec?

Run `sudo cscli hub update` to re-download a valid index

failed to parse index: %w

Error message

failed to parse index: %w

What it means

parseIndex fails when the hub index file content is not valid JSON for the expected items map structure, reported by json.Unmarshal. The file exists and is readable, but its contents are corrupt or in an unexpected format.

Source

Thrown at pkg/cwhub/hub.go:78

func (h *Hub) Load() error {
	h.logger.Debugf("loading hub idx %s", h.local.HubIndexFile)

	if err := h.parseIndex(); err != nil {
		return fmt.Errorf("invalid hub index: %w. Run 'sudo cscli hub update' to download the index again", err)
	}

	return h.localSync()
}

// parseIndex takes the content of an index file and fills the map of associated parsers/scenarios/collections.
func (h *Hub) parseIndex() error {
	bidx, err := os.ReadFile(h.local.HubIndexFile)
	if err != nil {
		return fmt.Errorf("unable to read index file: %w", err)
	}

	if err := json.Unmarshal(bidx, &h.items); err != nil {
		return fmt.Errorf("failed to parse index: %w", err)
	}

	// Iterate over the different types to complete the struct
	for _, itemType := range ItemTypes {
		for name, item := range h.GetItemMap(itemType) {
			if item == nil {
				// likely defined as empty object or null in the index file
				return fmt.Errorf("%s:%s has no index metadata", itemType, name)
			}

			if item.RemotePath == "" {
				return fmt.Errorf("%s:%s has no download path", itemType, name)
			}

			if (itemType == PARSERS || itemType == POSTOVERFLOWS) && item.Stage == "" {
				return fmt.Errorf("%s:%s has no stage", itemType, name)
			}

View on GitHub (pinned to 909b515798)

Solutions

Run `sudo cscli hub update` to re-download a valid index
Delete the corrupt file and run `cscli hub update` again: `rm /var/lib/crowdsec/data/hub/.index.json`
Validate the file with `jq . <file>` to see the exact JSON error and location
Check proxy/firewall is not injecting content into hub downloads

Defensive patterns

Strategy: validation

Validate before calling

bidx, err := os.ReadFile(indexFile)
if err == nil {
    var probe map[string]json.RawMessage
    if err := json.Unmarshal(bidx, &probe); err != nil {
        return fmt.Errorf("index is corrupt (%v); run 'cscli hub update'", err)
    }
}

Try / catch

if err := hub.Load(); err != nil {
    if strings.Contains(err.Error(), "failed to parse index") {
        logger.Error("corrupt hub index; removing and re-downloading")
        os.Remove(indexFile)
        if uerr := runHubUpdate(ctx); uerr != nil { return uerr }
        return hub.Load()
    }
    return err
}

Prevention

Don't hand-edit .index.json; regenerate it via `cscli hub update`
Ensure downloads complete atomically (the downloader uses temp files — don't bypass it)
Check that proxies don't inject HTML into hub downloads

When it happens

Trigger: Hub.Load → parseIndex when .index.json is truncated by an interrupted download, hand-edited with syntax errors, or written by an incompatible hub/cscli version.

Common situations: Interrupted `cscli hub update` (partial file), disk corruption, manual modification, or an index fetched through a proxy that returned an HTML error page instead of JSON.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/d6d20929d0301b71. Report an issue: GitHub.