{"record":{"id":"5ef303c5b39c533e","repo":"666ghj/MiroFish","slug":"generated-svg-must-be-canonical-utf-8","errorCode":null,"errorMessage":"generated SVG must be canonical UTF-8","messagePattern":"generated SVG must be canonical UTF-8","errorType":"exception","errorClass":"StarHistoryError","httpStatus":null,"severity":"error","filePath":"scripts/star_history.py","lineNumber":1076,"sourceCode":"            return None\n        for marker in (0xC0, 0xC1, 0xC2, 0xC3, 0xC5, 0xC6, 0xC7, 0xC9, 0xCA, 0xCB, 0xCD, 0xCE, 0xCF):\n            offset = payload.find(bytes((0xFF, marker)))\n            if offset >= 0 and offset + 9 <= len(payload):\n                return (\n                    int.from_bytes(payload[offset + 7 : offset + 9], \"big\"),\n                    int.from_bytes(payload[offset + 5 : offset + 7], \"big\"),\n                )\n        return None\n    return None\n\n\ndef _validate_svg(payload: bytes) -> None:\n    try:\n        decoded_payload = payload.decode(\"utf-8\", errors=\"strict\")\n    except UnicodeDecodeError as exc:\n        raise StarHistoryError(\"generated SVG must be strict UTF-8\") from exc\n    if decoded_payload.startswith(\"\\ufeff\") or \"\\x00\" in decoded_payload:\n        raise StarHistoryError(\"generated SVG must be canonical UTF-8\")\n    upper_payload = decoded_payload.upper()\n    if (\n        \"<!DOCTYPE\" in upper_payload\n        or \"<!ENTITY\" in upper_payload\n        or \"<?\" in decoded_payload\n    ):\n        raise StarHistoryError(\"generated SVG contains forbidden XML directives\")\n    try:\n        root = ET.fromstring(decoded_payload)\n    except ET.ParseError as exc:\n        raise StarHistoryError(\"generated SVG is not valid XML\") from exc\n    svg_namespace = \"{http://www.w3.org/2000/svg}\"\n    if root.tag != f\"{svg_namespace}svg\":\n        raise StarHistoryError(\"generated SVG root is invalid\")\n    allowed_attributes: dict[str, set[str]] = {\n        \"svg\": {\n            \"viewBox\",\n            \"width\",","sourceCodeStart":1058,"sourceCodeEnd":1094,"githubUrl":"https://github.com/666ghj/MiroFish/blob/b5b53acc57189a4a42e44a23e149dc655c98fe82/scripts/star_history.py#L1058-L1094","documentation":"Raised by `_validate_svg` (scripts/star_history.py:1076) when the payload decodes as UTF-8 but is not in canonical form: it starts with a UTF-8 BOM (U+FEFF, ef bb bf) or contains a NUL character (\\x00). BOMs confuse some XML consumers and NUL bytes are never legal in XML text, so the validator rejects both before parsing.","triggerScenarios":"SVG bytes produced/prefixed with `\\xef\\xbb\\xbf` (many Windows editors add a BOM on save), or containing embedded `\\x00` from a truncation/concatenation bug or binary splice. Fires when `decoded_payload.startswith(\"\\ufeff\") or \"\\x00\" in decoded_payload`.","commonSituations":"Editing the generated SVG in BOM-adding editors (old Notepad, some IDEs) and re-validating; cat-ing partial binary data into the file; forks that splice data-URI bytes into the payload and accidentally include NULs.","solutions":["Strip a leading BOM before validating/publishing: `payload = payload.lstrip(b\"\\xef\\xbb\\xbf\")` — better, save editors as 'UTF-8 without BOM'.","Find the NUL source: `payload.find(b\"\\x00\")` gives the offset; usually a wrong slice index when splicing bytes.","Do not post-process the SVG with binary tools; regenerate from `render_svg` instead.","Keep the renderer's data-URI constants intact — they are Base64 and BOM/NUL-free."],"exampleFix":"# before: re-validating a BOM-prefixed file\npayload = pathlib.Path(\"out.svg\").read_bytes()\n_validate_svg(payload)  # raises 'must be canonical UTF-8'\n\n# after: strip BOM, then validate\npayload = pathlib.Path(\"out.svg\").read_bytes()\npayload = payload.removeprefix(b\"\\xef\\xbb\\xbf\")\n_validate_svg(payload)","handlingStrategy":"try-catch","validationCode":"def is_canonical_utf8(payload: bytes) -> bool:\n    return not payload.startswith(b\"\\xef\\xbb\\xbf\") and b\"\\x00\" not in payload\n\nassert is_canonical_utf8(payload)","typeGuard":null,"tryCatchPattern":"try:\n    _validate_svg(payload)\nexcept StarHistoryError as exc:\n    if \"canonical UTF-8\" in str(exc):\n        payload = payload.removeprefix(b\"\\xef\\xbb\\xbf\").replace(b\"\\x00\", b\"\")\n        _validate_svg(payload)  # re-check after cleanup\n    else:\n        raise","preventionTips":["Save SVG files as 'UTF-8 without BOM'; check editor settings on Windows.","Audit byte-splicing code for NULs with payload.find(b'\\x00').","Prefer regenerating from render_svg over repairing edited files."],"tags":["svg","utf-8","bom","encoding","self-check"],"backgroundTag":null,"analyzedSha":"b5b53acc57189a4a42e44a23e149dc655c98fe82","analyzedAt":"2026-08-14T22:29:33.146Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}