{"record":{"id":"038c7def5d06c826","repo":"unslothai/unsloth","slug":"chat-template-contains-unpaired-surrogate-characte-038c7d","errorCode":null,"errorMessage":"Chat template contains unpaired surrogate characters.","messagePattern":"Chat template contains unpaired surrogate characters\\.","errorType":"validation","errorClass":"ValueError","httpStatus":422,"severity":"warning","filePath":"studio/backend/routes/settings.py","lineNumber":711,"sourceCode":"    gpu_layers: Optional[int] = Field(default = None, ge = -1, le = 1024)\n    n_cpu_moe: Optional[int] = Field(default = None, ge = 0, le = 1024)\n    gpu_ids: Optional[list[int]] = Field(default = None, max_length = MAX_GPU_IDS)\n    # An all-default save carries no fields, like a forget; None keeps the legacy contract.\n    remove: Optional[bool] = None\n    # Fill in, don't replace: the backfill reads the map once then writes each model, so another\n    # tab's save was overwritten by this browser's older copy. Field level, not entry level: a\n    # legacy entry holds only some fields, and skipping it would strand the rest.\n    fill_absent_fields: bool = False\n\n    @field_validator(\"chat_template_override\")\n    @classmethod\n    def _limit_chat_template_bytes(cls, value: Optional[str]) -> Optional[str]:\n        # Mirrors LoadRequest.normalize_blank_chat_template_override.\n        if value is None:\n            return None\n        size = chat_template_byte_length(value)\n        if size is None:\n            raise ValueError(\"Chat template contains unpaired surrogate characters.\")\n        if size > MAX_CHAT_TEMPLATE_BYTES:\n            raise ValueError(f\"Chat template exceeds the {MAX_CHAT_TEMPLATE_BYTES}-byte limit.\")\n        return value\n\n    @field_validator(\n        \"max_seq_length\",\n        \"custom_context_length\",\n        \"spec_draft_n_max\",\n        \"n_parallel\",\n        \"n_batch\",\n        \"n_ubatch\",\n        \"gpu_layers\",\n        \"n_cpu_moe\",\n        \"gpu_ids\",\n        mode = \"before\",\n    )\n    @classmethod\n    def _no_booleans(cls, value: Any) -> Any:","sourceCodeStart":693,"sourceCodeEnd":729,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/settings.py#L693-L729","documentation":"Pydantic validation failure (HTTP 422) from the chat_template_override validator on the model-overrides payload. It calls chat_template_byte_length(value), which returns None when the Python string contains unpaired UTF-16 surrogate code points (e.g. lone �) that have no valid UTF-8 encoding, so the byte size cannot be computed. Such strings typically enter via JSON payloads decoded as escaped surrogate pairs.","triggerScenarios":"PUT /settings/openai-auto-switch/overrides (or the model-overrides endpoint) with chat_template_override containing a lone escaped surrogate such as \"\\ud83d\" (half of an emoji) produced by bad string slicing or a broken encoder.","commonSituations":"Client-side string slicing that cuts an emoji in half (JS .slice on UTF-16 code units) then submits the remainder; hand-crafted JSON with \\uD800 escapes; data passed through a layer that encodes with errors='surrogatepass' and then round-trips.","solutions":["Fix the producer: never slice by UTF-16 code units; use Array.from(str) or code-point-aware operations before truncating.","Sanitize before submit: replace lone surrogates or validate with TextEncoder round-trip in JS.","If the template came from a file, re-save it as clean UTF-8 without surrogate escapes."],"exampleFix":"// before (JS)\nconst tpl = fullTemplate.slice(0, 50000); // may split a surrogate pair -> 422\nawait api.put(url, { chat_template_override: tpl });\n\n// after\nconst tpl = Array.from(fullTemplate).slice(0, 50000).join('');\nnew TextEncoder().encode(tpl); // throws on lone surrogate in old engines; use as canary\nawait api.put(url, { chat_template_override: tpl });","handlingStrategy":"validation","validationCode":"function hasLoneSurrogate(s: string): boolean {\n  for (let i = 0; i < s.length; i++) {\n    const c = s.charCodeAt(i);\n    if (c >= 0xD800 && c <= 0xDBFF && (i + 1 >= s.length || s.charCodeAt(i + 1) < 0xDC00)) return true;\n    if (c >= 0xDC00 && c <= 0xDFFF && (i === 0 || s.charCodeAt(i - 1) > 0xDBFF)) return true;\n  }\n  return false;\n}\nif (hasLoneSurrogate(tpl)) throw new Error('Template has unpaired surrogates');","typeGuard":"function isCleanTemplate(v: unknown): v is string {\n  return typeof v === 'string' && !hasLoneSurrogate(v);\n}","tryCatchPattern":"try { await api.put(url, { chat_template_override: tpl }); }\ncatch (e) {\n  if (e.status === 422 && /surrogate/i.test(e.detail?.toString() ?? '')) { recomputeTemplateFromSource(); return; }\n  throw e;\n}","preventionTips":["Truncate by code points (Array.from), never by UTF-16 code units.","Round-trip templates through TextEncoder/TextDecoder before storing.","Keep templates in UTF-8 files rather than embedded JSON string literals."],"tags":["pydantic","validation","http-422","unicode","surrogates","chat-template"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}