crowdsecurity/crowdsec · error

while unmarshalling allowlist item: %s

Error message

while unmarshalling allowlist item: %s

What it means

updateOneAllowlist pulls a community allowlist over HTTP from the URL given in the AllowlistLink and expects each response line to be a JSON object unmarshalled into models.AllowlistItem. This error wraps the json.Unmarshal failure for a single line, so the allowlist payload contained a line that is not valid JSON or not shaped like an AllowlistItem. It aborts the whole allowlist sync for that link.

Source

Thrown at pkg/apiserver/apic.go:753

	if err != nil {
		return fmt.Errorf("while pulling allowlist: %s", err)
	}

	resp, err := client.GetClient().Do(req)
	if err != nil {
		return fmt.Errorf("while pulling allowlist: %s", err)
	}
	defer resp.Body.Close()

	scanner := bufio.NewScanner(resp.Body)
	items := make([]*models.AllowlistItem, 0)

	for scanner.Scan() {
		item := scanner.Text()
		j := &models.AllowlistItem{}

		if err := json.Unmarshal([]byte(item), j); err != nil {
			return fmt.Errorf("while unmarshalling allowlist item: %s", err)
		}

		items = append(items, j)
	}

	list, err := a.dbClient.GetAllowListByID(ctx, *link.ID, false)
	if err != nil {
		if !ent.IsNotFound(err) {
			return fmt.Errorf("while getting allowlist %s: %s", *link.Name, err)
		}
	}

	if list == nil {
		list, err = a.dbClient.CreateAllowList(ctx, *link.Name, description, *link.ID, true)
		if err != nil {
			return fmt.Errorf("while creating allowlist %s: %s", *link.Name, err)
		}
	}

View on GitHub (pinned to 909b515798)

Solutions

1. Check the allowlist URL returns raw JSONL: curl -s <url> | head and inspect the first lines
2. Verify each line is a JSON object like {"value":"1.2.3.4"} matching models.AllowlistItem
3. Check that no HTML error/interstitial page is being served (status 200 with HTML body, e.g. Cloudflare challenge)
4. If self-hosting the allowlist, regenerate it in the expected JSONL format or upgrade crowdsec if the upstream format changed
5. Look at the wrapped %s message for the exact JSON offset/syntax problem

Example fix

// before: unmarshalling every line blindly
if err := json.Unmarshal([]byte(item), j); err != nil {
    return fmt.Errorf("while unmarshalling allowlist item: %s", err)
}
// after: skip blank/comment lines that aren't JSON items
item = strings.TrimSpace(item)
if item == "" || strings.HasPrefix(item, "#") {
    continue
}
if err := json.Unmarshal([]byte(item), j); err != nil {
    return fmt.Errorf("while unmarshalling allowlist item: %s", err)
}

// validate the pulled payload before syncing
for scanner.Scan() {
    line := strings.TrimSpace(scanner.Text())
    if line == "" || strings.HasPrefix(line, "#") {
        continue
    }
    if !json.Valid([]byte(line)) {
        log.Warnf("skipping non-JSON allowlist line: %q", line)
        continue
    }
    if !strings.HasPrefix(line, "{") {
        log.Warnf("skipping non-object allowlist line: %q", line)
        continue
    }
    // ... unmarshal as usual
}

if err := a.updateOneAllowlist(ctx, client, link); err != nil {
    var uerr *json.UnmarshalTypeError
    if errors.As(err, nil) && strings.Contains(err.Error(), "unmarshalling allowlist item") {
        log.Warnf("allowlist payload invalid, skipping sync for %s: %s", *link.Name, err)
        continue
    }
    log.Errorf("updating allowlists from CAPI: %s", err)
}

When it happens

Trigger: A line fetched from the allowlist URL is not valid JSON (e.g. an HTML error page, an empty or blank line, a truncated response) or lacks the required fields/types of AllowlistItem (e.g. a bare IP string instead of a {"value":...} object).

Common situations: The allowlist URL points to an error page or CDN block page instead of the raw JSONL file; the upstream allowlist format changed; a custom/private allowlist served via a web server mixes plain-text entries with JSON; the HTTP GET was redirected to a login or 404 page that still returned 200.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• unable to get allowlists: %w
• blocklist URL is nil
• pull already in progress
• no header in message, skipping
• no source user in header message, skipping

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/7f72827c43e33a7d. Report an issue: GitHub.