toon-format/toon · error · TypeError

Cannot encode containing an unpaired surrogate U+ at index

Error message

Cannot encode ${context} containing an unpaired surrogate U+${code.toString(16).toUpperCase()} at index ${index}

What it means

A TypeError (not ToonDecodeError) thrown by assertNoLoneSurrogate during value normalization. JavaScript strings can contain lone UTF-16 surrogate code units (U+D800–U+DFFF) that are not part of a valid surrogate pair; such strings cannot be encoded as valid UTF-8 TOON output, so encoding fails fast with the code unit and index. It is raised while encoding string values via normalizeValue.

Solutions

Fix the data source so the string contains only well-formed Unicode (use String.prototype.isWellFormed() / toWellFormed() to sanitize)
Avoid cutting surrogate pairs: iterate with for...of or Intl.Segmenter instead of index-based slice
Replace or strip the offending code unit at the reported index before encoding

Example fix

// before
const label = raw.slice(0, 5) // may split a surrogate pair
encoder.encode({ label })
// after
const label = raw.toWellFormed().slice(0, 5)
encoder.encode({ label })

Defensive patterns

Strategy: validation

Validate before calling

for (const [key, value] of Object.entries(data)) {
  if (typeof value === 'string' && !value.isWellFormed()) throw new Error(`lone surrogate in ${key}`)
}

Type guard

function isWellFormedString(v: unknown): v is string { return typeof v === 'string' && v.isWellFormed() }

Try / catch

try { encode(data) } catch (e) { if (e instanceof TypeError && e.message.includes('unpaired surrogate')) { const m = e.message.match(/U\+([0-9A-F]+) at index (\d+)/); console.error('bad surrogate', m?.[1], 'at index', m?.[2]); data = sanitize(data) } }

Prevention

Call String.prototype.toWellFormed() on external string input before encoding
Slice strings with code-point-aware methods (for...of, [...str], Intl.Segmenter), not raw charCodeAt indices
Beware of splitting emojis/CJK characters with slice/substring
Validate Unicode well-formedness at API boundaries

When it happens

Trigger: Calling TOON encode with a string value (or an object/array containing one) that includes an unpaired high or low surrogate — typically from slicing a string in the middle of a surrogate pair, decoding invalid bytes, or a single half of an emoji being isolated.

Common situations: String.slice/substring/substr cutting through an emoji or non-BMP character; binary data decoded with utf8 replacement gone wrong; data received from databases or APIs containing corrupted Unicode; regex operations on code-unit indices.

Related errors

AI-assisted analysis of toon-format/toon@604eac266e (2026-08-31). Data as JSON: /api/errors/19a1aa9113c1bb40. Report an issue: GitHub.

Appendix: source

Thrown at packages/toon/src/encode/normalize.ts:113

function assertNoLoneSurrogate(value: string, context: string): void {
  if (!SURROGATE_PATTERN.test(value)) {
    return
  }

  for (let index = 0; index < value.length; index++) {
    const code = value.charCodeAt(index)
    if (code < 0xD800 || code > 0xDFFF) {
      continue
    }

    const isHighSurrogate = code <= 0xDBFF
    const next = value.charCodeAt(index + 1)
    if (isHighSurrogate && next >= 0xDC00 && next <= 0xDFFF) {
      index++
      continue
    }

    throw new TypeError(
      `Cannot encode ${context} containing an unpaired surrogate U+${code.toString(16).toUpperCase()} at index ${index}`,
    )
  }
}

// #endregion

// #region Type guards

export function isJsonPrimitive(value: unknown): value is JsonPrimitive {
  return (
    value === null
    || typeof value === 'string'
    || typeof value === 'number'
    || typeof value === 'boolean'
  )
}

View on GitHub (pinned to 604eac266e)