ruvnet/ruflo · error · Error

canonical JSON does not support lone UTF-16 surrogates

Error message

canonical JSON does not support lone UTF-16 surrogates

What it means

assertUnicodeScalarString scans UTF-16 code units before canonical JSON encoding (and therefore before any digest derived from it). This throw site fires when a high surrogate (U+D800-U+DBFF, the first half of an astral character like an emoji) is not followed by a low surrogate — a split character. Digests over ill-formed strings are not stable across encodings, so canonicalization refuses them.

Source

Thrown at v3/@claude-flow/codex/src/harness/repository-state.ts:95

interface ContentState {
  baseCommit: string;
  treeId: string;
  trackedPatch: ContentDigest;
  untrackedManifest: UntrackedManifest;
  submodules: readonly SubmoduleState[];
}

function digest(value: string | Buffer): string {
  return `${SHA256_PREFIX}${createHash('sha256').update(value).digest('hex')}`;
}

function assertUnicodeScalarString(value: string): void {
  for (let index = 0; index < value.length; index += 1) {
    const unit = value.charCodeAt(index);
    if (unit >= 0xd800 && unit <= 0xdbff) {
      const next = value.charCodeAt(index + 1);
      if (!(next >= 0xdc00 && next <= 0xdfff)) {
        throw new Error('canonical JSON does not support lone UTF-16 surrogates');
      }
      index += 1;
    } else if (unit >= 0xdc00 && unit <= 0xdfff) {
      throw new Error('canonical JSON does not support lone UTF-16 surrogates');
    }
  }
}

function codeUnitCompare(left: string, right: string): number {
  return left < right ? -1 : left > right ? 1 : 0;
}

/** Recursive, locale-independent canonical JSON for JSON-safe contract values. */
export function canonicalJson(value: unknown): string {
  const ancestors = new Set<object>();
  const encode = (item: unknown): string => {
    if (item === null || typeof item === 'boolean') return JSON.stringify(item);
    if (typeof item === 'string') {

View on GitHub (pinned to fa13ee4ad6)

Solutions

1. Sanitize strings before hashing: s = s.toWellFormed() (ES2024) or replace lone surrogates with U+FFFD
2. Slice by code points, not UTF-16 indices: Array.from(str).slice(0, n).join('')
3. Pre-validate with a well-formedness check (String#isWellFormed or a manual scan) before calling snapshot/record APIs

Example fix

// before: cuts emoji in half, leaving a lone high surrogate
const label = text.substring(0, 40);

// after: cut on code-point boundaries and repair any ill-formed input
const label = Array.from(text.toWellFormed()).slice(0, 40).join('');

function isUnicodeScalarString(value: string): boolean {
  for (let i = 0; i < value.length; i += 1) {
    const unit = value.charCodeAt(i);
    if (unit >= 0xd800 && unit <= 0xdbff) {
      const next = value.charCodeAt(i + 1);
      if (!(next >= 0xdc00 && next <= 0xdfff)) return false;
      i += 1;
    } else if (unit >= 0xdc00 && unit <= 0xdfff) return false;
  }
  return true;
}

function isWellFormedString(value: unknown): value is string {
  return typeof value === 'string' && (value.isWellFormed?.() ?? isUnicodeScalarString(value));
}

try {
  return canonicalJson(payload);
} catch (error) {
  if (error instanceof Error && error.message === 'canonical JSON does not support lone UTF-16 surrogates') {
    return canonicalJson(sanitizeSurrogates(payload)); // JSON.stringify replacer using toWellFormed()
  }
  throw error;
}

When it happens

Trigger: canonicalJson (directly or via source-state snapshots / recordRun evidence) over strings produced by str.split('') and rejoin, substring/slice cutting inside an emoji, JSON.parse of payloads containing escaped lone surrogates like "\ud800", or strings assembled via charCodeAt/fromCharCode arithmetic.

Common situations: Truncating user display names, commit messages, or chat text to a fixed UTF-16 width; naive ellipsis logic on emoji-rich content; data ingested from sources that emit escaped lone surrogates in JSON.

Related errors

• canonical JSON requires finite, safe, non-negative-zero numb
• canonical JSON does not support undefined
• canonical JSON does not support ${typeof item}
• canonical JSON does not support cycles
• canonical JSON supports only arrays and plain objects

AI-assisted analysis of ruvnet/ruflo@fa13ee4ad6 (2026-08-18). Data as JSON: /api/errors/7ec4cff71712f812. Report an issue: GitHub.