666ghj/MiroFish · error · StarHistoryError

generated SVG must be canonical UTF-8

Error message

generated SVG must be canonical UTF-8

What it means

Raised by `_validate_svg` (scripts/star_history.py:1076) when the payload decodes as UTF-8 but is not in canonical form: it starts with a UTF-8 BOM (U+FEFF, ef bb bf) or contains a NUL character (\x00). BOMs confuse some XML consumers and NUL bytes are never legal in XML text, so the validator rejects both before parsing.

Source

Thrown at scripts/star_history.py:1076

            return None
        for marker in (0xC0, 0xC1, 0xC2, 0xC3, 0xC5, 0xC6, 0xC7, 0xC9, 0xCA, 0xCB, 0xCD, 0xCE, 0xCF):
            offset = payload.find(bytes((0xFF, marker)))
            if offset >= 0 and offset + 9 <= len(payload):
                return (
                    int.from_bytes(payload[offset + 7 : offset + 9], "big"),
                    int.from_bytes(payload[offset + 5 : offset + 7], "big"),
                )
        return None
    return None


def _validate_svg(payload: bytes) -> None:
    try:
        decoded_payload = payload.decode("utf-8", errors="strict")
    except UnicodeDecodeError as exc:
        raise StarHistoryError("generated SVG must be strict UTF-8") from exc
    if decoded_payload.startswith("\ufeff") or "\x00" in decoded_payload:
        raise StarHistoryError("generated SVG must be canonical UTF-8")
    upper_payload = decoded_payload.upper()
    if (
        "<!DOCTYPE" in upper_payload
        or "<!ENTITY" in upper_payload
        or "<?" in decoded_payload
    ):
        raise StarHistoryError("generated SVG contains forbidden XML directives")
    try:
        root = ET.fromstring(decoded_payload)
    except ET.ParseError as exc:
        raise StarHistoryError("generated SVG is not valid XML") from exc
    svg_namespace = "{http://www.w3.org/2000/svg}"
    if root.tag != f"{svg_namespace}svg":
        raise StarHistoryError("generated SVG root is invalid")
    allowed_attributes: dict[str, set[str]] = {
        "svg": {
            "viewBox",
            "width",

View on GitHub (pinned to b5b53acc57)

Solutions

  1. Strip a leading BOM before validating/publishing: `payload = payload.lstrip(b"\xef\xbb\xbf")` — better, save editors as 'UTF-8 without BOM'.
  2. Find the NUL source: `payload.find(b"\x00")` gives the offset; usually a wrong slice index when splicing bytes.
  3. Do not post-process the SVG with binary tools; regenerate from `render_svg` instead.
  4. Keep the renderer's data-URI constants intact — they are Base64 and BOM/NUL-free.

Example fix

# before: re-validating a BOM-prefixed file
payload = pathlib.Path("out.svg").read_bytes()
_validate_svg(payload)  # raises 'must be canonical UTF-8'

# after: strip BOM, then validate
payload = pathlib.Path("out.svg").read_bytes()
payload = payload.removeprefix(b"\xef\xbb\xbf")
_validate_svg(payload)
Defensive patterns

Strategy: try-catch

Validate before calling

def is_canonical_utf8(payload: bytes) -> bool:
    return not payload.startswith(b"\xef\xbb\xbf") and b"\x00" not in payload

assert is_canonical_utf8(payload)

Try / catch

try:
    _validate_svg(payload)
except StarHistoryError as exc:
    if "canonical UTF-8" in str(exc):
        payload = payload.removeprefix(b"\xef\xbb\xbf").replace(b"\x00", b"")
        _validate_svg(payload)  # re-check after cleanup
    else:
        raise

Prevention

When it happens

Trigger: SVG bytes produced/prefixed with `\xef\xbb\xbf` (many Windows editors add a BOM on save), or containing embedded `\x00` from a truncation/concatenation bug or binary splice. Fires when `decoded_payload.startswith("\ufeff") or "\x00" in decoded_payload`.

Common situations: Editing the generated SVG in BOM-adding editors (old Notepad, some IDEs) and re-validating; cat-ing partial binary data into the file; forks that splice data-URI bytes into the payload and accidentally include NULs.

Related errors

AI-assisted analysis of 666ghj/MiroFish@b5b53acc57 (2026-08-14). Data as JSON: /api/errors/5ef303c5b39c533e. Report an issue: GitHub.