astrid-runtime/astrid · error

WinFsp callback probe correlation mismatch

Error message

WinFsp callback probe correlation mismatch

What it means

probe_callback correlates responses with requests via request_id. If the response's request_id differs from the probe request's id, the reply is rejected — it is stale, duplicated, or belongs to another probe.

Solutions

1. Fix the responder to echo the exact request_id from the incoming probe into the response.
2. Give each daemon its own callback_path (derive from lease) so responses cannot cross-contaminate.
3. Purge stale queued responses on the callback socket before probing.
4. Ensure the probe is not retried against a socket with in-flight old replies.

Example fix

// before (responder)
StorageFilesystemResponseV2 { request_id: fixed_id(), .. }
// after
StorageFilesystemResponseV2 { request_id: request.request_id.clone(), .. }

fn response_correlates(req: &StorageFilesystemRequest, resp: &StorageFilesystemResponseV2) -> bool {
    resp.request_id == req.request_id
}

fn is_reply_to(req: &StorageFilesystemRequest, resp: &StorageFilesystemResponseV2) -> bool {
    resp.request_id == req.request_id
}

match probe_callback(&launch, &request).await {
    Err(e) if e.to_string().contains("correlation mismatch") => {
        // drain stale replies from the shared socket, then re-probe with a fresh request_id
        Err(e)
    },
    other => other,
}

When it happens

Trigger: run_private_service -> probe_callback when the responder echoes a request_id other than the one in the outgoing request: replayed responses from a prior probe, shared callback_path serving multiple daemons, or a responder that doesn't copy the id field.

Common situations: Multiple mount daemons sharing one callback socket so answers interleave; a persistent responder caching and replaying old responses; a responder implementation bug that hardcodes or regenerates request_id instead of echoing it.

Related errors

• WinFsp callback probe protocol mismatch
• WinFsp callback response exceeds limit
• WinFsp daemon returned an invalid stop acknowledgement
• build WinFsp callback filesystem
• daemon closed the response stream before the final marker

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/70d651afc346b104. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-storage-provider-winfsp/src/win.rs:369

    let length = u32::try_from(bytes.len()).context("WinFsp callback probe is too large")?;
    stream.write_all(&length.to_be_bytes()).await?;
    stream.write_all(&bytes).await?;
    stream.flush().await?;
    let mut response_length = [0_u8; 4];
    stream.read_exact(&mut response_length).await?;
    let length = u32::from_be_bytes(response_length) as usize;
    if length == 0 || length > SERVICE_MAX_CALLBACK_BYTES {
        bail!("WinFsp callback response exceeds limit");
    }
    let mut response_bytes = vec![0_u8; length];
    stream.read_exact(&mut response_bytes).await?;
    let response: StorageFilesystemResponseV2 =
        serde_json::from_slice(&response_bytes).context("decode WinFsp callback probe")?;
    if response.protocol_version != STORAGE_FILESYSTEM_PROTOCOL_V2 {
        bail!("WinFsp callback probe protocol mismatch");
    }
    if response.request_id != request.request_id {
        bail!("WinFsp callback probe correlation mismatch");
    }
    match response.outcome {
        StorageFilesystemOutcomeV2::Success(_) => Ok(()),
        StorageFilesystemOutcomeV2::Failure(StorageFilesystemFailureV1 { code, message }) => {
            bail!("WinFsp callback probe failed [{code}]: {message}")
        },
    }
}

async fn private_service_loop(
    filesystem: FileSystem,
    listener: local_transport::LocalListener,
    launch: &StorageProviderServiceLaunchV1,
) -> Result<()> {
    let mut filesystem = Some(filesystem);
    let mut poll = tokio::time::interval(SERVICE_POLL);
    loop {
        tokio::select! {

View on GitHub (pinned to affd8760f4)