unslothai/unsloth · warning · ValueError

Chat template contains unpaired surrogate characters.

Error message

Chat template contains unpaired surrogate characters.

What it means

Pydantic validation failure (HTTP 422) from the chat_template_override validator on the model-overrides payload. It calls chat_template_byte_length(value), which returns None when the Python string contains unpaired UTF-16 surrogate code points (e.g. lone �) that have no valid UTF-8 encoding, so the byte size cannot be computed. Such strings typically enter via JSON payloads decoded as escaped surrogate pairs.

Source

Thrown at studio/backend/routes/settings.py:711

    gpu_layers: Optional[int] = Field(default = None, ge = -1, le = 1024)
    n_cpu_moe: Optional[int] = Field(default = None, ge = 0, le = 1024)
    gpu_ids: Optional[list[int]] = Field(default = None, max_length = MAX_GPU_IDS)
    # An all-default save carries no fields, like a forget; None keeps the legacy contract.
    remove: Optional[bool] = None
    # Fill in, don't replace: the backfill reads the map once then writes each model, so another
    # tab's save was overwritten by this browser's older copy. Field level, not entry level: a
    # legacy entry holds only some fields, and skipping it would strand the rest.
    fill_absent_fields: bool = False

    @field_validator("chat_template_override")
    @classmethod
    def _limit_chat_template_bytes(cls, value: Optional[str]) -> Optional[str]:
        # Mirrors LoadRequest.normalize_blank_chat_template_override.
        if value is None:
            return None
        size = chat_template_byte_length(value)
        if size is None:
            raise ValueError("Chat template contains unpaired surrogate characters.")
        if size > MAX_CHAT_TEMPLATE_BYTES:
            raise ValueError(f"Chat template exceeds the {MAX_CHAT_TEMPLATE_BYTES}-byte limit.")
        return value

    @field_validator(
        "max_seq_length",
        "custom_context_length",
        "spec_draft_n_max",
        "n_parallel",
        "n_batch",
        "n_ubatch",
        "gpu_layers",
        "n_cpu_moe",
        "gpu_ids",
        mode = "before",
    )
    @classmethod
    def _no_booleans(cls, value: Any) -> Any:

View on GitHub (pinned to 203007d190)

Solutions

1. Fix the producer: never slice by UTF-16 code units; use Array.from(str) or code-point-aware operations before truncating.
2. Sanitize before submit: replace lone surrogates or validate with TextEncoder round-trip in JS.
3. If the template came from a file, re-save it as clean UTF-8 without surrogate escapes.

Example fix

// before (JS)
const tpl = fullTemplate.slice(0, 50000); // may split a surrogate pair -> 422
await api.put(url, { chat_template_override: tpl });

// after
const tpl = Array.from(fullTemplate).slice(0, 50000).join('');
new TextEncoder().encode(tpl); // throws on lone surrogate in old engines; use as canary
await api.put(url, { chat_template_override: tpl });

function hasLoneSurrogate(s: string): boolean {
  for (let i = 0; i < s.length; i++) {
    const c = s.charCodeAt(i);
    if (c >= 0xD800 && c <= 0xDBFF && (i + 1 >= s.length || s.charCodeAt(i + 1) < 0xDC00)) return true;
    if (c >= 0xDC00 && c <= 0xDFFF && (i === 0 || s.charCodeAt(i - 1) > 0xDBFF)) return true;
  }
  return false;
}
if (hasLoneSurrogate(tpl)) throw new Error('Template has unpaired surrogates');

function isCleanTemplate(v: unknown): v is string {
  return typeof v === 'string' && !hasLoneSurrogate(v);
}

try { await api.put(url, { chat_template_override: tpl }); }
catch (e) {
  if (e.status === 422 && /surrogate/i.test(e.detail?.toString() ?? '')) { recomputeTemplateFromSource(); return; }
  throw e;
}

When it happens

Trigger: PUT /settings/openai-auto-switch/overrides (or the model-overrides endpoint) with chat_template_override containing a lone escaped surrogate such as "\ud83d" (half of an emoji) produced by bad string slicing or a broken encoder.

Common situations: Client-side string slicing that cuts an emoji in half (JS .slice on UTF-16 code units) then submits the remainder; hand-crafted JSON with \uD800 escapes; data passed through a layer that encodes with errors='surrogatepass' and then round-trips.

Related errors

• Chat template contains unpaired surrogate characters.
• Chat template exceeds the {MAX_CHAT_TEMPLATE_BYTES}-byte lim
• Provide either content_base64 or file_ids, not both
• Provide either content_base64 or file_ids
• file_ids must not be empty

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/038c7def5d06c826. Report an issue: GitHub.