headroomlabs-ai/headroom · error · ValueError

SmartCrusher: invalid protected_patterns regex {p!r}: {e}

Error message

SmartCrusher: invalid protected_patterns regex {p!r}: {e}

What it means

SmartCrusher's audit-safe mode compiles each entry of `protected_patterns` with `re.compile`; a pattern that raises `re.error` is re-raised as ValueError naming the offending pattern. Invalid regexes are treated as caller bugs, not swallowed, because silently treating a broken pattern as "no protection" would leave rows the caller believes are protected actually unprotected.

Source

Thrown at headroom/transforms/smart_crusher.py:561

    # survive the compressed output (never dropped, never marker-only).

    @staticmethod
    def _compile_protected_patterns(patterns: list[str] | None) -> list[re.Pattern[str]]:
        """Compile `protected_patterns` once at construction time.

        A pattern that fails to compile is a caller bug, not something
        to swallow — silently treating an invalid regex as "no rows
        protected" would defeat the entire point of audit-safe mode
        (rows the caller believes are protected wouldn't be).
        """
        if not patterns:
            return []
        compiled = []
        for p in patterns:
            try:
                compiled.append(re.compile(p))
            except re.error as e:
                raise ValueError(
                    f"SmartCrusher: invalid protected_patterns regex {p!r}: {e}"
                ) from e
        return compiled

    @staticmethod
    def _canon(item: Any) -> str:
        """Canonical JSON text for a row.

        Used both for protected-pattern matching and for identity
        comparison across the crush boundary — kept rows are
        re-serialized by Rust, so rows are matched by content, not by
        Python object identity.
        """
        return json.dumps(item, sort_keys=True, default=str)

    def _row_matches_protected(self, item: Any) -> bool:
        text = self._canon(item)
        return any(p.search(text) for p in self._protected_patterns)

View on GitHub (pinned to 322425c43b)

Solutions

1. Fix the named pattern — the message includes the exact pattern and the underlying `re.error` reason
2. Test every pattern standalone before passing it: `re.compile(p)` in a startup check or config loader
3. When a pattern must match literal text (filenames, IDs), use `re.escape(value)` instead of interpolating raw user input into a regex

Example fix

# before
patterns = [f"^client:{client_id}$"]  # client_id='acme(1)' -> re.error? no, but metachars bite
patterns = ["rows[client"]            # unbalanced bracket -> re.error

# after
import re
patterns = [re.escape(f"client:{client_id}")]
re.compile(patterns[0])  # validate at config-load time, not mid-compression

import re

def compile_protected_patterns(patterns: list[str]) -> list[re.Pattern]:
    compiled = []
    for p in patterns or []:
        try:
            compiled.append(re.compile(p))
        except re.error as e:
            raise ValueError(f"invalid protected pattern {p!r}: {e}") from e
    return compiled

compiled = compile_protected_patterns(patterns)  # before SmartCrusher call
crusher = SmartCrusher(protected_patterns=patterns)

def patterns_compile(patterns: list[str]) -> bool:
    return all(_compiles(p) for p in patterns)

def _compiles(p: str) -> bool:
    try:
        re.compile(p); return True
    except re.error:
        return False

try:
    result = crusher.compress(items, protected_patterns=patterns)
except ValueError as e:
    if "invalid protected_patterns regex" in str(e):
        bad = patterns  # message names the pattern; fix source of patterns and abort
        raise
    raise

When it happens

Trigger: Passing `protected_patterns=["[unclosed"]`, `"*leading"`, or any string that fails Python `re.compile` (unbalanced brackets, invalid quantifier placement, bad group syntax) to SmartCrusher compression with protection enabled.

Common situations: User-supplied or config-file regexes never validated upstream; patterns written for a different regex flavor (e.g. JS/PCRE lookarounds or possessive quantifiers unsupported by `re`); escaping bugs when building patterns dynamically from strings containing regex metacharacters.

Related errors

• unknown compaction format {resolved_format!r}; expected one
• invalid pipeline config TOML: {0}
• recommendations file not found: {0}
• bedrock_eventstream_parse_failed
• items_json must be JSON: {e}

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/3bf475274a6cc031. Report an issue: GitHub.