{"record":{"id":"99d4fd1a54e596b6","repo":"t8y2/dbx","slug":"invalid-unicode-codepoint-value-x","errorCode":null,"errorMessage":"invalid Unicode codepoint: {value:#X}","messagePattern":"invalid Unicode codepoint: (.+?)","errorType":"validation","errorClass":"rusqlite::Error::UserFunctionError","httpStatus":null,"severity":"error","filePath":"crates/dbx-core/src/db/sqlite.rs","lineNumber":495,"sourceCode":"\n    Ok(result)\n}\n\nfn sqlite_unistr_codepoint(chars: &[char], start: usize, digits: usize) -> rusqlite::Result<Option<char>> {\n    if start + digits > chars.len() {\n        return Ok(None);\n    }\n\n    let mut value = 0_u32;\n    for ch in &chars[start..start + digits] {\n        let Some(digit) = ch.to_digit(16) else {\n            return Ok(None);\n        };\n        value = (value << 4) | digit;\n    }\n\n    std::char::from_u32(value)\n        .map(Some)\n        .ok_or_else(|| sqlite_function_error(format!(\"invalid Unicode codepoint: {value:#X}\")))\n}\n\nfn sqlite_function_error(message: impl Into<String>) -> rusqlite::Error {\n    rusqlite::Error::UserFunctionError(Box::new(std::io::Error::new(std::io::ErrorKind::InvalidInput, message.into())))\n}\n\npub fn path_has_sqlite_header(path: &Path) -> Result<bool, String> {\n    let mut file = std::fs::File::open(path).map_err(|e| format!(\"failed to open file: {e}\"))?;\n    let mut header = [0_u8; 16];\n    match file.read_exact(&mut header) {\n        Ok(()) => Ok(&header == SQLITE_DATABASE_HEADER),\n        Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => Ok(false),\n        Err(e) => Err(format!(\"failed to read file header: {e}\")),\n    }\n}\n\nfn validate_existing_sqlite_file(path: &str) -> Result<(), String> {","sourceCodeStart":477,"sourceCodeEnd":513,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/crates/dbx-core/src/db/sqlite.rs#L477-L513","documentation":"sqlite_unistr_codepoint parses the numeric value of a \\XXXX escape for the SQLite UNISTR() user function and converts it with std::char::from_u32. If the accumulated value is not a valid Unicode scalar value (e.g. surrogates U+D800–U+DFFF or values above U+10FFFF), the function raises 'invalid Unicode codepoint: 0xVALUE' wrapped as a rusqlite UserFunctionError.","triggerScenarios":"Calling SQL UNISTR() with a codepoint escape that decodes to a surrogate or out-of-range value, e.g. UNISTR('\\D800') or UNISTR('\\110000').","commonSituations":"Hand-written escape sequences intended as literal text (not Unicode escapes), data migrated from systems using surrogate pairs encoded individually, or typos in hex escapes producing values over 0x10FFFF.","solutions":["Replace surrogate codepoints with the actual character or a proper pair of valid escapes encoding the combined codepoint.","Use a codepoint within U+0000–U+10FFFF excluding U+D800–U+DFFF.","Escape a literal backslash if the sequence was not meant to be a Unicode escape.","Encode characters outside the BMP as a single codepoint value (e.g. \\1F600) rather than surrogate halves."],"exampleFix":"-- before\nSELECT UNISTR('\\D83D\\DE00');  -- surrogate halves\n-- after\nSELECT UNISTR('\\1F600');  -- single valid codepoint","handlingStrategy":"validation","validationCode":"-- Pre-check escapes in application code before running UNISTR:\n-- valid codepoints: 0x0..=0x10FFFF, excluding 0xD800..=0xDFFF\n-- e.g. reject UNISTR('\\D800') in input validation","typeGuard":"// Rust\nfn is_valid_scalar(v: u32) -> bool {\n    std::char::from_u32(v).is_some()\n}","tryCatchPattern":"// Rust\nmatch conn.query_row(sql, [], |r| ...) {\n    Err(rusqlite::Error::UserFunctionError(e)) if e.to_string().contains(\"invalid Unicode codepoint\") => {\n        // sanitize the \\XXXX escapes in `sql` and retry\n    }\n    other => other,\n}","preventionTips":["Validate UNISTR arguments against valid Unicode scalar ranges before executing.","Never encode surrogate halves separately; use a single codepoint for astral characters.","Escape literal backslashes when the text is not meant to be a Unicode escape."],"tags":["sqlite","unicode","user-function"],"backgroundTag":"invalid-unicode-codepoint","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}