{"record":{"id":"7f72827c43e33a7d","repo":"crowdsecurity/crowdsec","slug":"while-unmarshalling-allowlist-item-s","errorCode":null,"errorMessage":"while unmarshalling allowlist item: %s","messagePattern":"while unmarshalling allowlist item: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/apiserver/apic.go","lineNumber":753,"sourceCode":"\tif err != nil {\n\t\treturn fmt.Errorf(\"while pulling allowlist: %s\", err)\n\t}\n\n\tresp, err := client.GetClient().Do(req)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"while pulling allowlist: %s\", err)\n\t}\n\tdefer resp.Body.Close()\n\n\tscanner := bufio.NewScanner(resp.Body)\n\titems := make([]*models.AllowlistItem, 0)\n\n\tfor scanner.Scan() {\n\t\titem := scanner.Text()\n\t\tj := &models.AllowlistItem{}\n\n\t\tif err := json.Unmarshal([]byte(item), j); err != nil {\n\t\t\treturn fmt.Errorf(\"while unmarshalling allowlist item: %s\", err)\n\t\t}\n\n\t\titems = append(items, j)\n\t}\n\n\tlist, err := a.dbClient.GetAllowListByID(ctx, *link.ID, false)\n\tif err != nil {\n\t\tif !ent.IsNotFound(err) {\n\t\t\treturn fmt.Errorf(\"while getting allowlist %s: %s\", *link.Name, err)\n\t\t}\n\t}\n\n\tif list == nil {\n\t\tlist, err = a.dbClient.CreateAllowList(ctx, *link.Name, description, *link.ID, true)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"while creating allowlist %s: %s\", *link.Name, err)\n\t\t}\n\t}","sourceCodeStart":735,"sourceCodeEnd":771,"githubUrl":"https://github.com/crowdsecurity/crowdsec/blob/909b5157986a2b2c2163300fdaef5ed01289f7d2/pkg/apiserver/apic.go#L735-L771","documentation":"updateOneAllowlist pulls a community allowlist over HTTP from the URL given in the AllowlistLink and expects each response line to be a JSON object unmarshalled into models.AllowlistItem. This error wraps the json.Unmarshal failure for a single line, so the allowlist payload contained a line that is not valid JSON or not shaped like an AllowlistItem. It aborts the whole allowlist sync for that link.","triggerScenarios":"A line fetched from the allowlist URL is not valid JSON (e.g. an HTML error page, an empty or blank line, a truncated response) or lacks the required fields/types of AllowlistItem (e.g. a bare IP string instead of a {\"value\":...} object).","commonSituations":"The allowlist URL points to an error page or CDN block page instead of the raw JSONL file; the upstream allowlist format changed; a custom/private allowlist served via a web server mixes plain-text entries with JSON; the HTTP GET was redirected to a login or 404 page that still returned 200.","solutions":["Check the allowlist URL returns raw JSONL: curl -s <url> | head and inspect the first lines","Verify each line is a JSON object like {\"value\":\"1.2.3.4\"} matching models.AllowlistItem","Check that no HTML error/interstitial page is being served (status 200 with HTML body, e.g. Cloudflare challenge)","If self-hosting the allowlist, regenerate it in the expected JSONL format or upgrade crowdsec if the upstream format changed","Look at the wrapped %s message for the exact JSON offset/syntax problem"],"exampleFix":"// before: unmarshalling every line blindly\nif err := json.Unmarshal([]byte(item), j); err != nil {\n    return fmt.Errorf(\"while unmarshalling allowlist item: %s\", err)\n}\n// after: skip blank/comment lines that aren't JSON items\nitem = strings.TrimSpace(item)\nif item == \"\" || strings.HasPrefix(item, \"#\") {\n    continue\n}\nif err := json.Unmarshal([]byte(item), j); err != nil {\n    return fmt.Errorf(\"while unmarshalling allowlist item: %s\", err)\n}","handlingStrategy":"validation","validationCode":"// validate the pulled payload before syncing\nfor scanner.Scan() {\n    line := strings.TrimSpace(scanner.Text())\n    if line == \"\" || strings.HasPrefix(line, \"#\") {\n        continue\n    }\n    if !json.Valid([]byte(line)) {\n        log.Warnf(\"skipping non-JSON allowlist line: %q\", line)\n        continue\n    }\n    if !strings.HasPrefix(line, \"{\") {\n        log.Warnf(\"skipping non-object allowlist line: %q\", line)\n        continue\n    }\n    // ... unmarshal as usual\n}","typeGuard":null,"tryCatchPattern":"if err := a.updateOneAllowlist(ctx, client, link); err != nil {\n    var uerr *json.UnmarshalTypeError\n    if errors.As(err, nil) && strings.Contains(err.Error(), \"unmarshalling allowlist item\") {\n        log.Warnf(\"allowlist payload invalid, skipping sync for %s: %s\", *link.Name, err)\n        continue\n    }\n    log.Errorf(\"updating allowlists from CAPI: %s\", err)\n}","preventionTips":["Serve allowlists as raw JSONL, never through a page that can return HTML (check the URL, disable CDN challenges)","curl the allowlist URL and validate every line is a JSON object before pointing crowdsec at it","Skip/handle empty and comment lines in generated allowlist files","Pin/check the upstream allowlist format version when self-hosting"],"tags":["json","allowlist","unmarshal","capi"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"909b5157986a2b2c2163300fdaef5ed01289f7d2","analyzedAt":"2026-09-06T12:27:26.012Z","contentChangedAt":"2026-09-06T12:27:26.012Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}