vectordotdev/vector · error

{}

Error message

{}

What it means

The disk buffer checkpointer loads its checkpoints from a JSON file. This error is raised when serde_json fails to deserialize the file's contents into the checkpoint map; the serde error is wrapped into an io::Error with ErrorKind::InvalidData. The message text is the underlying serde_json error (the "{}" template).

Solutions

1. Restore the checkpoints file from backup or remove/rename it to let the buffer re-initialize (accepting re-delivery of buffered data).
2. Inspect the file's JSON with a validator to find the exact parse error location.
3. Recreate the affected disk buffer directory if the data can be discarded.

Example fix

// before (corrupt checkpoints.json)
{"last_position": "abc"}
// after
{"last_position": 1024}

// validate checkpoint file before the buffer reads it
fn checkpoint_file_valid(path: &Path) -> bool {
    std::fs::read(path)
        .map(|b| serde_json::from_slice::<serde_json::Value>(&b).is_ok())
        .unwrap_or(false)
}

match checkpointer.read_checkpoints().await {
    Err(e) if e.kind() == io::ErrorKind::InvalidData => {
        // corrupt checkpoints: rename aside and reinitialize
        fs::rename(path, path.with_extension("corrupt")).ok();
        Default::default()
    }
    other => other?,
}

When it happens

Trigger: The checkpoints file on disk is corrupt, truncated, or was written by an incompatible format/version; manual edits or a crash mid-write left invalid JSON; the file is empty or contains wrong types (e.g. a non-u64 where a sequence number is expected).

Common situations: Host crash or disk-full during checkpoint write; running a newer/older Vector against an old data directory; someone editing or replacing the buffer directory contents by hand.

Related errors

• cannot extend a file through the truncation API
• InvalidInput
• a record with a next ID must have an event count
• cannot extend a file through the truncation API
• cant ever be empty

AI-assisted analysis of vectordotdev/vector@0d4ab78a4f (2026-09-16). Data as JSON: /api/errors/2d3dd5fd6a8e9d30. Report an issue: GitHub.

Appendix: source

Thrown at lib/file-source-common/src/checkpointer.rs:289

            Err(error) if error.kind() == io::ErrorKind::NotFound => {
                // This is expected, so no warning needed
            }
            Err(error) => {
                warn!(message = "Unable to load checkpoint data.", %error);
            }
        }
    }

    async fn read_checkpoints_file(&self, path: &Path) -> Result<State, io::Error> {
        // Possible optimization: mmap the file into a slice and pass it into serde_json instead of
        // calling read_to_end. Need to investigate if this would work with tokio::fs::File

        let mut reader = BufReader::new(File::open(path).await?);
        let mut output = Vec::new();
        reader.read_to_end(&mut output).await?;

        serde_json::from_slice(&output[..])
            .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))
    }
}

#[cfg(test)]
mod test {
    use chrono::{Duration, Utc};
    use similar_asserts::assert_eq;
    use tempfile::tempdir;
    use tokio::fs;

    use super::{
        CHECKPOINT_FILE_NAME, Checkpoint, Checkpointer, FileFingerprint, FilePosition,
        TMP_FILE_NAME,
    };

    #[test]
    fn test_checkpointer_basics() {
        let fingerprints = vec![

View on GitHub (pinned to 0d4ab78a4f)