astrid-runtime/astrid · error

FSKit callback probe correlation mismatch

Error message

FSKit callback probe correlation mismatch

What it means

The FSKit storage provider sends a callback probe over a local Unix socket with a request_id and expects a response carrying the same id. If the response's request_id differs from the one sent, the transport is returning answers for a different request, so the provider aborts the probe rather than misattributing an outcome. This guards against cross-request response confusion in the multiplexed callback channel.

Solutions

1. Verify the FSKit extension echoes the exact request_id from StorageFilesystemRequestV2 in every response
2. Ensure each probe uses a fresh socket/connection so responses cannot interleave
3. Check that request.request_id is unique per probe (e.g. monotonically increasing) and not reused
4. Retry the probe once; if mismatches persist, log both ids and report the extension version

Example fix

// before: reusing one stream for concurrent probes
let response = read_callback_response(&mut shared_stream).await?;
// after: give each probe its own connection
let (mut stream, _) = listener.accept().await?;
let response = read_callback_response(&mut stream).await?;
assert_eq!(response.request_id, request.request_id);

fn probe_ids_match(sent: &StorageFilesystemRequestV2, resp: &StorageFilesystemResponseV2) -> bool { resp.request_id == sent.request_id }

match probe_callback(&req).await { Err(e) if e.to_string().contains("correlation mismatch") => retry_probe_once(&req).await, other => other }

When it happens

Trigger: probe_callback (called from run) writes a length-prefixed JSON probe then calls read_callback_response; the returned StorageFilesystemResponseV2 has a request_id != request.request_id — i.e. the responder echoed a stale or wrong id, responses were reordered/interleaved, or a previous probe's late answer was consumed by this read.

Common situations: A buggy or older FSKit extension that doesn't echo request ids; two probes racing on the same socket; a responder that crashed and a leftover buffered response being drained.

Related errors

• daemon closed the response stream before the final marker
• daemon rejected status request
• daemon returned an unexpected status response
• detached FUSE service exceeded the startup response size
• FSKit callback probe failed

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/4072a2d77618262e. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-storage-provider-fskit/src/service.rs:232

    let mut stream = local_transport::connect(&launch.lease.callback_path)
        .await
        .context("connect FSKit lease callback")?;
    let request = StorageFilesystemRequestV2 {
        protocol_version: STORAGE_FILESYSTEM_PROTOCOL_V2,
        request_id: format!("fskit-service-{}", launch.lease.mount_id),
        lease_token: launch.lease.lease_token.clone(),
        operation: StorageFilesystemOperationV2::Stat {
            path: String::new(),
        },
    };
    let bytes = serde_json::to_vec(&request).context("encode FSKit callback probe")?;
    let length = u32::try_from(bytes.len()).context("FSKit callback probe is too large")?;
    stream.write_all(&length.to_be_bytes()).await?;
    stream.write_all(&bytes).await?;
    stream.flush().await?;
    let response = read_callback_response(&mut stream).await?;
    if response.request_id != request.request_id {
        bail!("FSKit callback probe correlation mismatch");
    }
    match response.outcome {
        StorageFilesystemOutcomeV2::Success(_) => Ok(()),
        StorageFilesystemOutcomeV2::Failure(StorageFilesystemFailureV1 { code, message }) => {
            bail!("FSKit callback probe failed [{code}]: {message}")
        },
    }
}

async fn read_callback_response(stream: &mut LocalStream) -> Result<StorageFilesystemResponseV2> {
    let mut length = [0_u8; 4];
    stream.read_exact(&mut length).await?;
    let length = u32::from_be_bytes(length) as usize;
    if length == 0 || length > MAX_CALLBACK_BYTES {
        bail!("FSKit callback response exceeds the bounded frame size");
    }
    let mut bytes = vec![0_u8; length];
    stream.read_exact(&mut bytes).await?;

View on GitHub (pinned to affd8760f4)