unslothai/unsloth · error · ValueError

Chat template contains unpaired surrogate characters.

Error message

Chat template contains unpaired surrogate characters.

What it means

Field validator on ValidateChatTemplateRequest.template that first computes the template's byte length via chat_template_byte_length. That helper returns None when the string cannot be UTF-8 encoded — which happens when it contains unpaired surrogate code points (U+D800–U+DFFF), typically introduced by decoding bytes with surrogateescape or by copy-pasting from a lossy source. The validator raises so the malformed template is rejected before it can crash Jinja rendering later.

Source

Thrown at studio/backend/picker/schemas.py:35

    JSON can carry an unpaired surrogate, as a truncated emoji paste produces.
    json decodes it fine and .encode("utf-8") then raises. Callers treat None as
    "reject": such a template can never render.
    """
    try:
        return len(value.encode("utf-8"))
    except UnicodeEncodeError:
        return None


class ValidateChatTemplateRequest(BaseModel):
    template: str = Field(default = "")

    @field_validator("template")
    @classmethod
    def _enforce_template_size(cls, value: str) -> str:
        size = chat_template_byte_length(value)
        if size is None:
            raise ValueError("Chat template contains unpaired surrogate characters.")
        if size > MAX_CHAT_TEMPLATE_BYTES:
            raise ValueError(f"Chat template exceeds the {MAX_CHAT_TEMPLATE_BYTES}-byte limit.")
        return value


class ValidateChatTemplateResponse(BaseModel):
    valid: bool
    error: Optional[str] = None


class ModelTemplateResponse(BaseModel):
    model_name: str
    chat_template: Optional[str] = None

View on GitHub (pinned to 203007d190)

Solutions

Re-read the template source with strict UTF-8 (encoding='utf-8', default errors) instead of surrogateescape, or fix the underlying bytes.
Sanitize the string before sending: s.encode('utf-8', 'strict') in a try/except, or s.encode('utf-8','replace').decode('utf-8') to strip/replace the surrogates.
If the template came from a JSON file, validate the file with a strict JSON parser and remove any \ud800-\udfff escape sequences that are not part of a valid pair.

Example fix

# before
template = open('tpl.j2', errors='surrogateescape').read()  # may contain lone surrogates
req = ValidateChatTemplateRequest(template=template)

# after
template = open('tpl.j2', encoding='utf-8').read()  # strict UTF-8
req = ValidateChatTemplateRequest(template=template)

Defensive patterns

Strategy: validation

Validate before calling

def is_clean_utf8(template: str) -> bool:
    try:
        template.encode("utf-8")
        return True
    except UnicodeEncodeError:
        return False

def sanitize(template: str) -> str:
    return template.encode("utf-8", "replace").decode("utf-8")

Try / catch

try:
    resp = client.validate_chat_template(ValidateChatTemplateRequest(template=tpl))
except ValidationError as e:
    if "unpaired surrogate" in str(e):
        tpl = tpl.encode("utf-8", "replace").decode("utf-8")
        resp = client.validate_chat_template(ValidateChatTemplateRequest(template=tpl))
    else:
        raise

Prevention

Always read template files with encoding='utf-8' (strict); never surrogateescape.
Run template strings through a UTF-8 round-trip check before sending them to the API.

When it happens

Trigger: POSTing a chat template string containing lone surrogates, e.g. produced by open(path, errors='surrogateescape').read(), json.loads of malformed \udXXX escapes, or data round-tripped through a system that broke a surrogate pair apart.

Common situations: Loading templates from files with invalid UTF-8 using surrogateescape error handling; JSON payloads hand-built with escaped lone surrogates; templates concatenated from chunks that split a character (emoji/astral-plane) mid-sequence in Python 2-style code.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/2d71079f2677668d. Report an issue: GitHub.