{"record":{"id":"2d71079f2677668d","repo":"unslothai/unsloth","slug":"chat-template-contains-unpaired-surrogate-characte","errorCode":null,"errorMessage":"Chat template contains unpaired surrogate characters.","messagePattern":"Chat template contains unpaired surrogate characters\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/picker/schemas.py","lineNumber":35,"sourceCode":"    JSON can carry an unpaired surrogate, as a truncated emoji paste produces.\n    json decodes it fine and .encode(\"utf-8\") then raises. Callers treat None as\n    \"reject\": such a template can never render.\n    \"\"\"\n    try:\n        return len(value.encode(\"utf-8\"))\n    except UnicodeEncodeError:\n        return None\n\n\nclass ValidateChatTemplateRequest(BaseModel):\n    template: str = Field(default = \"\")\n\n    @field_validator(\"template\")\n    @classmethod\n    def _enforce_template_size(cls, value: str) -> str:\n        size = chat_template_byte_length(value)\n        if size is None:\n            raise ValueError(\"Chat template contains unpaired surrogate characters.\")\n        if size > MAX_CHAT_TEMPLATE_BYTES:\n            raise ValueError(f\"Chat template exceeds the {MAX_CHAT_TEMPLATE_BYTES}-byte limit.\")\n        return value\n\n\nclass ValidateChatTemplateResponse(BaseModel):\n    valid: bool\n    error: Optional[str] = None\n\n\nclass ModelTemplateResponse(BaseModel):\n    model_name: str\n    chat_template: Optional[str] = None\n","sourceCodeStart":17,"sourceCodeEnd":49,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/picker/schemas.py#L17-L49","documentation":"Field validator on ValidateChatTemplateRequest.template that first computes the template's byte length via chat_template_byte_length. That helper returns None when the string cannot be UTF-8 encoded — which happens when it contains unpaired surrogate code points (U+D800–U+DFFF), typically introduced by decoding bytes with surrogateescape or by copy-pasting from a lossy source. The validator raises so the malformed template is rejected before it can crash Jinja rendering later.","triggerScenarios":"POSTing a chat template string containing lone surrogates, e.g. produced by open(path, errors='surrogateescape').read(), json.loads of malformed \\udXXX escapes, or data round-tripped through a system that broke a surrogate pair apart.","commonSituations":"Loading templates from files with invalid UTF-8 using surrogateescape error handling; JSON payloads hand-built with escaped lone surrogates; templates concatenated from chunks that split a character (emoji/astral-plane) mid-sequence in Python 2-style code.","solutions":["Re-read the template source with strict UTF-8 (encoding='utf-8', default errors) instead of surrogateescape, or fix the underlying bytes.","Sanitize the string before sending: s.encode('utf-8', 'strict') in a try/except, or s.encode('utf-8','replace').decode('utf-8') to strip/replace the surrogates.","If the template came from a JSON file, validate the file with a strict JSON parser and remove any \\ud800-\\udfff escape sequences that are not part of a valid pair."],"exampleFix":"# before\ntemplate = open('tpl.j2', errors='surrogateescape').read()  # may contain lone surrogates\nreq = ValidateChatTemplateRequest(template=template)\n\n# after\ntemplate = open('tpl.j2', encoding='utf-8').read()  # strict UTF-8\nreq = ValidateChatTemplateRequest(template=template)","handlingStrategy":"validation","validationCode":"def is_clean_utf8(template: str) -> bool:\n    try:\n        template.encode(\"utf-8\")\n        return True\n    except UnicodeEncodeError:\n        return False\n\ndef sanitize(template: str) -> str:\n    return template.encode(\"utf-8\", \"replace\").decode(\"utf-8\")","typeGuard":null,"tryCatchPattern":"try:\n    resp = client.validate_chat_template(ValidateChatTemplateRequest(template=tpl))\nexcept ValidationError as e:\n    if \"unpaired surrogate\" in str(e):\n        tpl = tpl.encode(\"utf-8\", \"replace\").decode(\"utf-8\")\n        resp = client.validate_chat_template(ValidateChatTemplateRequest(template=tpl))\n    else:\n        raise","preventionTips":["Always read template files with encoding='utf-8' (strict); never surrogateescape.","Run template strings through a UTF-8 round-trip check before sending them to the API."],"tags":["pydantic","unicode","chat-template","validation"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}