t8y2/dbx · error · rusqlite::Error::UserFunctionError

invalid Unicode codepoint: {value:#X}

Error message

invalid Unicode codepoint: {value:#X}

What it means

sqlite_unistr_codepoint parses the numeric value of a \XXXX escape for the SQLite UNISTR() user function and converts it with std::char::from_u32. If the accumulated value is not a valid Unicode scalar value (e.g. surrogates U+D800–U+DFFF or values above U+10FFFF), the function raises 'invalid Unicode codepoint: 0xVALUE' wrapped as a rusqlite UserFunctionError.

Source

Thrown at crates/dbx-core/src/db/sqlite.rs:495

    Ok(result)
}

fn sqlite_unistr_codepoint(chars: &[char], start: usize, digits: usize) -> rusqlite::Result<Option<char>> {
    if start + digits > chars.len() {
        return Ok(None);
    }

    let mut value = 0_u32;
    for ch in &chars[start..start + digits] {
        let Some(digit) = ch.to_digit(16) else {
            return Ok(None);
        };
        value = (value << 4) | digit;
    }

    std::char::from_u32(value)
        .map(Some)
        .ok_or_else(|| sqlite_function_error(format!("invalid Unicode codepoint: {value:#X}")))
}

fn sqlite_function_error(message: impl Into<String>) -> rusqlite::Error {
    rusqlite::Error::UserFunctionError(Box::new(std::io::Error::new(std::io::ErrorKind::InvalidInput, message.into())))
}

pub fn path_has_sqlite_header(path: &Path) -> Result<bool, String> {
    let mut file = std::fs::File::open(path).map_err(|e| format!("failed to open file: {e}"))?;
    let mut header = [0_u8; 16];
    match file.read_exact(&mut header) {
        Ok(()) => Ok(&header == SQLITE_DATABASE_HEADER),
        Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => Ok(false),
        Err(e) => Err(format!("failed to read file header: {e}")),
    }
}

fn validate_existing_sqlite_file(path: &str) -> Result<(), String> {

View on GitHub (pinned to c0390bff16)

Solutions

1. Replace surrogate codepoints with the actual character or a proper pair of valid escapes encoding the combined codepoint.
2. Use a codepoint within U+0000–U+10FFFF excluding U+D800–U+DFFF.
3. Escape a literal backslash if the sequence was not meant to be a Unicode escape.
4. Encode characters outside the BMP as a single codepoint value (e.g. \1F600) rather than surrogate halves.

Example fix

-- before
SELECT UNISTR('\D83D\DE00');  -- surrogate halves
-- after
SELECT UNISTR('\1F600');  -- single valid codepoint

-- Pre-check escapes in application code before running UNISTR:
-- valid codepoints: 0x0..=0x10FFFF, excluding 0xD800..=0xDFFF
-- e.g. reject UNISTR('\D800') in input validation

// Rust
fn is_valid_scalar(v: u32) -> bool {
    std::char::from_u32(v).is_some()
}

// Rust
match conn.query_row(sql, [], |r| ...) {
    Err(rusqlite::Error::UserFunctionError(e)) if e.to_string().contains("invalid Unicode codepoint") => {
        // sanitize the \XXXX escapes in `sql` and retry
    }
    other => other,
}

When it happens

Trigger: Calling SQL UNISTR() with a codepoint escape that decodes to a surrogate or out-of-range value, e.g. UNISTR('\D800') or UNISTR('\110000').

Common situations: Hand-written escape sequences intended as literal text (not Unicode escapes), data migrated from systems using surrogate pairs encoded individually, or typos in hex escapes producing values over 0x10FFFF.

Related errors

• Failed to open storage
• Failed to open storage
• Failed to enable SQLite legacy_alter_table for the backup re
• SQLite worker has no open database
• SQL is empty

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/99d4fd1a54e596b6. Report an issue: GitHub.