crowdsecurity/crowdsec · error

invalid user_agent regex for bot entry '%s' in %s: %w

Error message

invalid user_agent regex for bot entry '%s' in %s: %w

What it means

botFileInit pre-compiles the optional user_agent field as a case-insensitive regular expression via compileBotRegex. This error is thrown when regexp.Compile rejects the pattern (unbalanced parens/brackets, dangling escape, invalid repetition, etc.). Compiling at load time catches bad regexes before any traffic is matched.

Source

Thrown at pkg/exprhelpers/botfile.go:70

	dec.DisallowUnknownFields()

	if err := dec.Decode(entry); err != nil {
		return fmt.Errorf("failed to parse JSON line in %s: %w", filename, err)
	}

	if entry.Name == "" {
		return fmt.Errorf("missing mandatory 'name' field in %s: %s", filename, line)
	}

	if len(entry.IPs)+len(entry.Ranges)+len(entry.RDNS) == 0 {
		return fmt.Errorf("bot entry '%s' in %s has no identity verification (need at least one of ips/ranges/rdns)", entry.Name, filename)
	}

	var err error

	if entry.UserAgent != "" {
		if entry.uaRegex, err = compileBotRegex(entry.UserAgent); err != nil {
			return fmt.Errorf("invalid user_agent regex for bot entry '%s' in %s: %w", entry.Name, filename, err)
		}
	}

	for _, p := range entry.Paths {
		re, err := compileBotRegex(p)
		if err != nil {
			return fmt.Errorf("invalid path regex '%s' for bot entry '%s' in %s: %w", p, entry.Name, filename, err)
		}

		entry.pathRegexes = append(entry.pathRegexes, re)
	}

	entry.ipSet = make(map[netip.Addr]struct{}, len(entry.IPs))

	for _, ip := range entry.IPs {
		addr, err := netip.ParseAddr(ip)
		if err != nil {
			return fmt.Errorf("invalid IP '%s' for bot entry '%s' in %s: %w", ip, entry.Name, filename, err)

View on GitHub (pinned to 909b515798)

Solutions

  1. Escape regex metacharacters in literal UA substrings, e.g. use "Mozilla/5\.0 \(compatible\)" not "Mozilla/5.0 (compatible)".
  2. Fix the syntax error reported by the wrapped regexp/syntax message (it names the offending position).
  3. Replace PCRE-only constructs (lookaheads, lookbehinds, backreferences) with RE2-compatible patterns.
  4. Simplify to a substring pattern if you do not need regex power: "Googlebot" matches anywhere (compile is case-insensitive).
  5. Test the pattern first with Go: regexp.Compile("(?i)" + pattern) in a scratch program.

Example fix

// before
{"name":"chrome","user_agent":"Chrome/(ver 1+2)"}
// after
{"name":"chrome","user_agent":"Chrome/\(ver 1\+2\)"}
Defensive patterns

Strategy: validation

Validate before calling

// compile-check the user_agent pattern exactly as the loader will
if _, err := regexp.Compile("(?i)" + uaPattern); err != nil {
	// reject before FileInit
}
valid := err == nil

Try / catch

if err := exprhelpers.FileInit(botFile, "bots"); err != nil {
	if strings.Contains(err.Error(), "invalid user_agent regex") {
		log.Errorf("fix the UA pattern: %v", err)
	}
	return err
}

Prevention

When it happens

Trigger: A bots JSONL entry has a "user_agent" value that is not a valid Go RE2 regex, e.g. "Mozilla*" (invalid repetition), "(bad" (unbalanced group), or a trailing backslash. The wrapped error is the regexp/syntax parse error.

Common situations: Copying UA strings verbatim that contain regex metacharacters like +, (, ), ? without escaping; hand-written regex with a typo; converting patterns from PCRE with unsupported constructs like lookaheads (?=...) or backreferences, which RE2 rejects.

Related errors

AI-assisted analysis of crowdsecurity/crowdsec@909b515798 (2026-09-06). Data as JSON: /api/errors/e70531b759a2bf12. Report an issue: GitHub.