What causes "failed to parse JSON line in %s: %w" in crowdsecurity/crowdsec?

Validate the line with a JSON linter or `echo ' ' | jq .` and fix the syntax error indicated by the wrapped %w error (it names the exact byte offset).

crowdsecurity/crowdsec · error

failed to parse JSON line in %s: %w

Error message

failed to parse JSON line in %s: %w

What it means

botFileInit parses each line of a 'bots' JSON-lines data file into a botEntry struct with DisallowUnknownFields enabled. This error is returned when the line is not valid JSON at all, or contains fields not defined on botEntry. It is thrown during data-file loading (FileInit) so that malformed bot data fails fast at startup instead of silently mis-matching at runtime.

Source

Thrown at pkg/exprhelpers/botfile.go:55

	uaRegex     *regexp.Regexp
	pathRegexes []*regexp.Regexp
	ipSet       map[netip.Addr]struct{}
	prefixes    []netip.Prefix
	rdnsRegexes []*regexp.Regexp
}

func compileBotRegex(pattern string) (*regexp.Regexp, error) {
	return regexp.Compile("(?i)" + pattern) // Force case insensitive match
}

func botFileInit(filename string, line string) error {
	entry := &botEntry{}

	dec := json.NewDecoder(strings.NewReader(line))
	dec.DisallowUnknownFields()

	if err := dec.Decode(entry); err != nil {
		return fmt.Errorf("failed to parse JSON line in %s: %w", filename, err)
	}

	if entry.Name == "" {
		return fmt.Errorf("missing mandatory 'name' field in %s: %s", filename, line)
	}

	if len(entry.IPs)+len(entry.Ranges)+len(entry.RDNS) == 0 {
		return fmt.Errorf("bot entry '%s' in %s has no identity verification (need at least one of ips/ranges/rdns)", entry.Name, filename)
	}

	var err error

	if entry.UserAgent != "" {
		if entry.uaRegex, err = compileBotRegex(entry.UserAgent); err != nil {
			return fmt.Errorf("invalid user_agent regex for bot entry '%s' in %s: %w", entry.Name, filename, err)
		}
	}

View on GitHub (pinned to 909b515798)

Solutions

Validate the line with a JSON linter or `echo '<line>' | jq .` and fix the syntax error indicated by the wrapped %w error (it names the exact byte offset).
Check for unknown field names: only name, user_agent, paths, ips, ranges, rdns are accepted — rename any other key or remove it.
Ensure the file is strict JSON-lines: one JSON object per line, no commas between lines, no BOM, no blank malformed lines.
If the file came from an external source or older CrowdSec version, re-download the current format or migrate field names.
Run `cscli` or the FileInit path again after fixing; the error message includes the filename so fix that specific file.

Example fix

// before (unknown field + trailing comma)
{"name":"mybot","ip":"1.2.3.4",}
// after
{"name":"mybot","ips":["1.2.3.4"]}

Defensive patterns

Strategy: validation

Validate before calling

// pre-validate a bot file line before handing it to the loader
func validBotLine(line string) bool {
	var raw map[string]json.RawMessage
	if err := json.Unmarshal([]byte(line), &raw); err != nil {
		return false
	}
	allowed := map[string]bool{"name": true, "user_agent": true, "paths": true, "ips": true, "ranges": true, "rdns": true}
	for k := range raw {
		if !allowed[k] {
			return false
		}
	}
	return raw["name"] != nil
}

Try / catch

if err := exprhelpers.FileInit(botFile, "bots"); err != nil {
	var lineno int
	// the loader reports the offending line/file; log and abort startup
	log.Fatalf("bot data file rejected: %v", err)
}

Prevention

Lint every data file with jq before deployment: `jq -c . file.jsonl` rejects invalid lines.
Validate field names against the documented schema (name, user_agent, paths, ips, ranges, rdns).
Generate entries programmatically with json.Marshal instead of string concatenation.
Strip BOM and CRLF when producing files on Windows.
Add a CI check that parses all bot data files with the same DisallowUnknownFields decoder.

When it happens

Trigger: A line in the bots data file is not valid JSON (trailing comma, single quotes, truncation, BOM), or uses a field name not in the botEntry schema (name, user_agent, paths, ips, ranges, rdns) — e.g. 'ip' instead of 'ips' or 'useragent' instead of 'user_agent'.

Common situations: Hand-edited or copy-pasted bot entries with JSON syntax typos; converting a CSV/YAML bot list to JSONL incorrectly; upgrading CrowdSec after a schema change and keeping an old file with renamed fields; a build/template step emitting two JSON objects on one line.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/31771648fd405c5f. Report an issue: GitHub.