{"record":{"id":"7ec4cff71712f812","repo":"ruvnet/ruflo","slug":"canonical-json-does-not-support-lone-utf-16-surrog","errorCode":null,"errorMessage":"canonical JSON does not support lone UTF-16 surrogates","messagePattern":"canonical JSON does not support lone UTF-16 surrogates","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"v3/@claude-flow/codex/src/harness/repository-state.ts","lineNumber":95,"sourceCode":"interface ContentState {\n  baseCommit: string;\n  treeId: string;\n  trackedPatch: ContentDigest;\n  untrackedManifest: UntrackedManifest;\n  submodules: readonly SubmoduleState[];\n}\n\nfunction digest(value: string | Buffer): string {\n  return `${SHA256_PREFIX}${createHash('sha256').update(value).digest('hex')}`;\n}\n\nfunction assertUnicodeScalarString(value: string): void {\n  for (let index = 0; index < value.length; index += 1) {\n    const unit = value.charCodeAt(index);\n    if (unit >= 0xd800 && unit <= 0xdbff) {\n      const next = value.charCodeAt(index + 1);\n      if (!(next >= 0xdc00 && next <= 0xdfff)) {\n        throw new Error('canonical JSON does not support lone UTF-16 surrogates');\n      }\n      index += 1;\n    } else if (unit >= 0xdc00 && unit <= 0xdfff) {\n      throw new Error('canonical JSON does not support lone UTF-16 surrogates');\n    }\n  }\n}\n\nfunction codeUnitCompare(left: string, right: string): number {\n  return left < right ? -1 : left > right ? 1 : 0;\n}\n\n/** Recursive, locale-independent canonical JSON for JSON-safe contract values. */\nexport function canonicalJson(value: unknown): string {\n  const ancestors = new Set<object>();\n  const encode = (item: unknown): string => {\n    if (item === null || typeof item === 'boolean') return JSON.stringify(item);\n    if (typeof item === 'string') {","sourceCodeStart":77,"sourceCodeEnd":113,"githubUrl":"https://github.com/ruvnet/ruflo/blob/fa13ee4ad60ac2090b1480656eb233521790d640/v3/@claude-flow/codex/src/harness/repository-state.ts#L77-L113","documentation":"assertUnicodeScalarString scans UTF-16 code units before canonical JSON encoding (and therefore before any digest derived from it). This throw site fires when a high surrogate (U+D800-U+DBFF, the first half of an astral character like an emoji) is not followed by a low surrogate — a split character. Digests over ill-formed strings are not stable across encodings, so canonicalization refuses them.","triggerScenarios":"canonicalJson (directly or via source-state snapshots / recordRun evidence) over strings produced by str.split('') and rejoin, substring/slice cutting inside an emoji, JSON.parse of payloads containing escaped lone surrogates like \"\\ud800\", or strings assembled via charCodeAt/fromCharCode arithmetic.","commonSituations":"Truncating user display names, commit messages, or chat text to a fixed UTF-16 width; naive ellipsis logic on emoji-rich content; data ingested from sources that emit escaped lone surrogates in JSON.","solutions":["Sanitize strings before hashing: s = s.toWellFormed() (ES2024) or replace lone surrogates with U+FFFD","Slice by code points, not UTF-16 indices: Array.from(str).slice(0, n).join('')","Pre-validate with a well-formedness check (String#isWellFormed or a manual scan) before calling snapshot/record APIs"],"exampleFix":"// before: cuts emoji in half, leaving a lone high surrogate\nconst label = text.substring(0, 40);\n\n// after: cut on code-point boundaries and repair any ill-formed input\nconst label = Array.from(text.toWellFormed()).slice(0, 40).join('');","handlingStrategy":"type-guard","validationCode":"function isUnicodeScalarString(value: string): boolean {\n  for (let i = 0; i < value.length; i += 1) {\n    const unit = value.charCodeAt(i);\n    if (unit >= 0xd800 && unit <= 0xdbff) {\n      const next = value.charCodeAt(i + 1);\n      if (!(next >= 0xdc00 && next <= 0xdfff)) return false;\n      i += 1;\n    } else if (unit >= 0xdc00 && unit <= 0xdfff) return false;\n  }\n  return true;\n}","typeGuard":"function isWellFormedString(value: unknown): value is string {\n  return typeof value === 'string' && (value.isWellFormed?.() ?? isUnicodeScalarString(value));\n}","tryCatchPattern":"try {\n  return canonicalJson(payload);\n} catch (error) {\n  if (error instanceof Error && error.message === 'canonical JSON does not support lone UTF-16 surrogates') {\n    return canonicalJson(sanitizeSurrogates(payload)); // JSON.stringify replacer using toWellFormed()\n  }\n  throw error;\n}","preventionTips":["Run .toWellFormed() on all externally sourced strings before they enter evidence","Slice and truncate by code points (Array.from(str).slice(n)), never by UTF-16 index","Add a fixture with emoji + combining marks to tests of anything that truncates text"],"tags":["unicode","utf-16","surrogate","canonical-json","serialization"],"backgroundTag":"lone-utf16-surrogate","analyzedSha":"fa13ee4ad60ac2090b1480656eb233521790d640","analyzedAt":"2026-08-18T21:34:22.708Z","schemaVersion":2},"datasetVersion":"2026-08-22T09:17:25.309Z"}