astrid-runtime/astrid · error

FUSE callback probe correlation mismatch

Error message

FUSE callback probe correlation mismatch

What it means

probe_callback sends a serialized request with a request_id over the local transport to the launch's callback socket, then reads a response that must carry the same request_id. If the response's request_id differs, the reply cannot be correlated with the probe (wrong peer, protocol desync, or a foreign service on the socket), and the launch probe fails.

Source

Thrown at crates/astrid-storage-provider-fuse/src/service.rs:316

    let mut stream = local_transport::connect(&launch.lease.callback_path)
        .await
        .context("connect FUSE lease callback")?;
    let request = StorageFilesystemRequestV2 {
        protocol_version: STORAGE_FILESYSTEM_PROTOCOL_V2,
        request_id: format!("fuse-service-{}", Uuid::new_v4()),
        lease_token: launch.lease.lease_token.clone(),
        operation: StorageFilesystemOperationV2::Stat {
            path: String::new(),
        },
    };
    let bytes = serde_json::to_vec(&request).context("encode FUSE callback probe")?;
    let length = u32::try_from(bytes.len()).context("FUSE callback probe is too large")?;
    stream.write_all(&length.to_be_bytes()).await?;
    stream.write_all(&bytes).await?;
    stream.flush().await?;
    let response = read_callback_response(&mut stream).await?;
    if response.request_id != request.request_id {
        bail!("FUSE callback probe correlation mismatch");
    }
    match response.outcome {
        StorageFilesystemOutcomeV2::Success(_) => Ok(()),
        StorageFilesystemOutcomeV2::Failure(StorageFilesystemFailureV1 { code, message }) => {
            bail!("FUSE callback probe failed [{code}]: {message}")
        },
    }
}

async fn read_callback_response(stream: &mut LocalStream) -> Result<StorageFilesystemResponseV2> {
    let mut length = [0_u8; 4];
    stream.read_exact(&mut length).await?;
    let length = u32::from_be_bytes(length) as usize;
    if length == 0 || length > MAX_CALLBACK_BYTES {
        bail!("FUSE callback response exceeds the bounded frame size");
    }
    let mut bytes = vec![0_u8; length];
    stream.read_exact(&mut bytes).await?;

View on GitHub (pinned to affd8760f4)

Solutions

1. Verify the process listening on launch.lease.callback_path is the current launch's peer (kill stale instances from earlier launches)
2. Ensure the callback peer echoes request.request_id verbatim in its StorageFilesystemOutcomeV2 response
3. Serialize probes over a given callback socket — never issue concurrent probes that share one connection/path without multiplexing
4. Check kernel/provider version compatibility for the callback protocol framing

Example fix

// before: peer responds with a fixed id
response.request_id = DEFAULT_ID;

// after: echo the caller's id
response.request_id = request.request_id;

if std::fs::read_dir(lease.resource_path.join("..")).is_err() {
    // lease directory gone/stale — relaunch before probing
}

fn is_probe_response(r: &CallbackResponse, want: u64) -> bool { r.request_id == want }

match probe_callback(launch).await {
    Err(e) if e.to_string().contains("correlation mismatch") => {
        // stale/foreign peer on callback socket: kill old instance and relaunch
    }
    Err(e) => return Err(e),
    Ok(()) => {}
}

When it happens

Trigger: Calling run_launch when the callback endpoint answers with a response whose request_id field does not equal request.request_id — e.g. the socket is served by a different/older process, responses are interleaved from concurrent probes, or the peer mishandles the id field.

Common situations: A leftover service from a previous launch still listening on the callback path and answering with its own ids; concurrent launches sharing a callback socket causing crossed responses; a peer implementation bug echoing the wrong request_id; protocol version mismatch between kernel and provider.

Related errors

• detached FUSE service exceeded the startup response size
• FUSE service control request exceeds limit
• opaque capsule assets cannot be symlinks: {}
• unexpected daemon response: {other:?}
• unexpected daemon response: {other:?}

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/805a5159dd7e6a0d. Report an issue: GitHub.