{"record":{"id":"51f78617e7fcae60","repo":"affaan-m/ECC","slug":"label-must-contain-valid-utf-8-text","errorCode":null,"errorMessage":"${label} must contain valid UTF-8 text.","messagePattern":"(.+?) must contain valid UTF-8 text\\.","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"scripts/lib/memory-vault-format.js","lineNumber":218,"sourceCode":"    updatedAt: validateTimestamp(memory.updatedAt, 'updated_at'),\n    body: normalizeBody(memory.body),\n  };\n}\n\nfunction serializeMemoryDocument(memory) {\n  const normalized = normalizeMemory(memory);\n  const metadata = FRONTMATTER_FIELDS.map(([serializedKey, objectKey]) => (\n    `${serializedKey}: ${JSON.stringify(normalized[objectKey])}`\n  )).join('\\n');\n  const body = normalized.body.length > 0 ? `\\n\\n${normalized.body}` : '';\n  return `---\\n${metadata}\\n---${body}\\n`;\n}\n\nfunction decodeUtf8(buffer, label = 'text') {\n  try {\n    return FATAL_UTF8_DECODER.decode(buffer);\n  } catch {\n    throw new Error(`${label} must contain valid UTF-8 text.`);\n  }\n}\n\nfunction parseFrontmatterLine(line, sourcePath, seen) {\n  const separator = line.indexOf(':');\n  if (separator <= 0) {\n    throw new Error(`Invalid memory frontmatter line in ${sourcePath}.`);\n  }\n  const serializedKey = line.slice(0, separator).trim();\n  const objectKey = FRONTMATTER_KEYS.get(serializedKey);\n  if (!objectKey) {\n    throw new Error(`Unknown memory frontmatter field in ${sourcePath}.`);\n  }\n  if (seen.has(objectKey)) {\n    throw new Error(`Duplicate memory frontmatter field in ${sourcePath}.`);\n  }\n  const rawValue = line.slice(separator + 1).trim();\n  try {","sourceCodeStart":200,"sourceCodeEnd":236,"githubUrl":"https://github.com/affaan-m/ECC/blob/01e15490f04e29cfefe3896951f43db46994d8ee/scripts/lib/memory-vault-format.js#L200-L236","documentation":"Thrown by decodeUtf8() when the fatal TextDecoder rejects the input buffer. The library uses TextDecoder('utf-8', { fatal: true }) (FATAL_UTF8_DECODER) rather than the lenient default, because memory documents are security-relevant state and silently substituting replacement characters (U+FFFD) could hide tampering or corruption. The label argument identifies what was being decoded (e.g. 'memory document').","triggerScenarios":"A memory .md file on disk has been corrupted by a partial write, a disk error, or an editor that saved in latin-1/cp1252 instead of UTF-8. A vault file was patched in place by a tool that injected raw bytes. readRegularTextFile() opens the file with O_NOFOLLOW and reads bytes, then decodeUtf8() converts — any non-shortest-form UTF-8 or invalid continuation byte triggers this.","commonSituations":"Power failure mid-write left a truncated multi-byte sequence. Windows editor saved as 'ANSI' instead of 'UTF-8'. Pipe through iconv with the wrong source encoding. Diff/patch tool that operated in byte mode and produced invalid sequences.","solutions":["Re-save the offending file as UTF-8 with a text editor (use iconv -f latin-1 -t utf-8 file or recode UTF-8 file).","Restore the file from version control or backup; treat non-UTF-8 vault contents as corruption.","Run doctorMemoryVault() to enumerate which files are unreadable — they appear in invalidFiles.","If you control the producer, ensure it writes with encoding: 'utf8' and never mixes Buffer concat from non-UTF-8 sources."],"exampleFix":"// before: file saved as cp1252, decode throws\nconst source = readRegularTextFile(filePath, { trustedRoot: root });\n\n// after: re-encode on disk first\n// shell: iconv -f cp1252 -t utf-8 file.md > file.md.utf8 && mv file.md.utf8 file.md\nconst source = readRegularTextFile(filePath, { trustedRoot: root });","handlingStrategy":"try-catch","validationCode":"const buffer = fs.readFileSync(filePath);\ntry {\n  new TextDecoder('utf-8', { fatal: true }).decode(buffer);\n} catch {\n  throw new Error(`file ${filePath} is not valid UTF-8; re-save before importing`);\n}","typeGuard":null,"tryCatchPattern":"try {\n  readRegularTextFile(filePath, { trustedRoot: root });\n} catch (err) {\n  if (/must contain valid UTF-8 text/.test(err.message)) {\n    // quarantine the file and log; do not fall back to lenient decode\n    throw new Error(`vault file ${filePath} is corrupt; restore from backup`);\n  }\n  throw err;\n}","preventionTips":["Ensure all producers write with encoding: 'utf8' (Node's default for writeFileSync of strings).","Ban editors configured for latin-1/cp1252 on vault directories.","After any tool modifies a vault file in place, re-open it once to confirm it still parses."],"tags":["encoding","utf-8","memory-vault","io"],"backgroundTag":null,"analyzedSha":"01e15490f04e29cfefe3896951f43db46994d8ee","analyzedAt":"2026-08-13T00:31:08.655Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}