invalid utf-8 sequence of

Error message

invalid utf-8 sequence of {N} bytes from index {M} (wrapped Utf8Error from pair-token store file)

What it means

The pair-token store file is read from disk and must be valid UTF-8; std::str::from_utf8 failure is wrapped in PairTokenStoreError::Io as InvalidData, surfacing Rust's native Utf8Error message ('invalid utf-8 sequence of N bytes from index M'). NotFound is tolerated (empty store) but genuinely corrupt bytes abort the load.

Solutions

Restore the pair-token store file from a backup or delete it to start with an empty store (tokens must be re-paired)
Re-save the file as UTF-8 if it was hand-edited with a wrong encoding
Check for concurrent writers and ensure writes are atomic (write-temp-then-rename)
Run fsck / check disk health if corruption recurs

Example fix

// before: hand-edited with latin-1 escapes
iconv -f ISO-8859-1 -t UTF-8 pair-tokens.toml -o pair-tokens.toml
// after: or reset
mv pair-tokens.toml pair-tokens.toml.bak  # store recreates empty file

Defensive patterns

Strategy: try-catch

Validate before calling

let bytes = std::fs::read(&store_path)?;
if std::str::from_utf8(&bytes).is_err() { eprintln!("pair-token store is not valid UTF-8"); }

Try / catch

match store.load() {
    Err(PairTokenStoreError::Io(e)) if e.kind() == std::io::ErrorKind::InvalidData && e.to_string().contains("utf-8") => {
        // restore backup or reset store and re-pair
    }
    other => other?,
}

Prevention

Write the store atomically (temp file + rename)
Edit only with UTF-8-capable editors
Check disk health if corruption recurs
Back up the store before upgrades

When it happens

Trigger: Calling PairTokenStore::load (→ load_from_disk) when the TOML file on disk contains invalid UTF-8 bytes.

Common situations: Partial/truncated writes from a crash; the file was edited with a binary tool or got encoding mangled; disk corruption; another process wrote bytes concurrently.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/b3606514a6586f2c. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-kernel/src/pair_token/mod.rs:504

        {
            let _ = &self.path;
            return Ok(Vec::new());
        }
        #[cfg(not(all(target_arch = "wasm32", target_os = "unknown")))]
        {
            self.load_from_disk()
        }
    }

    #[cfg(not(all(target_arch = "wasm32", target_os = "unknown")))]
    fn load_from_disk(&self) -> Result<Vec<PairToken>, PairTokenStoreError> {
        let bytes = match std::fs::read(&self.path) {
            Ok(b) => b,
            Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()),
            Err(e) => return Err(PairTokenStoreError::Io(e)),
        };
        let text = std::str::from_utf8(&bytes).map_err(|e| {
            PairTokenStoreError::Io(std::io::Error::new(std::io::ErrorKind::InvalidData, e))
        })?;
        if text.trim().is_empty() {
            if let Err(error) = self.save_to_disk(&[]) {
                warn!(
                    path = %self.path.display(),
                    %error,
                    "could not normalize empty pair-token store"
                );
            }
            return Ok(Vec::new());
        }
        let probe: SchemaProbe = toml::from_str(text).map_err(PairTokenStoreError::Toml)?;
        if probe.schema_version > STORE_SCHEMA_VERSION {
            return Err(PairTokenStoreError::Io(std::io::Error::new(
                std::io::ErrorKind::InvalidData,
                format!(
                    "pair-token store schema {} is newer than supported schema {STORE_SCHEMA_VERSION}",
                    probe.schema_version

View on GitHub (pinned to affd8760f4)