{"record":{"id":"805a5159dd7e6a0d","repo":"astrid-runtime/astrid","slug":"fuse-callback-probe-correlation-mismatch","errorCode":null,"errorMessage":"FUSE callback probe correlation mismatch","messagePattern":"FUSE callback probe correlation mismatch","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/astrid-storage-provider-fuse/src/service.rs","lineNumber":316,"sourceCode":"    let mut stream = local_transport::connect(&launch.lease.callback_path)\n        .await\n        .context(\"connect FUSE lease callback\")?;\n    let request = StorageFilesystemRequestV2 {\n        protocol_version: STORAGE_FILESYSTEM_PROTOCOL_V2,\n        request_id: format!(\"fuse-service-{}\", Uuid::new_v4()),\n        lease_token: launch.lease.lease_token.clone(),\n        operation: StorageFilesystemOperationV2::Stat {\n            path: String::new(),\n        },\n    };\n    let bytes = serde_json::to_vec(&request).context(\"encode FUSE callback probe\")?;\n    let length = u32::try_from(bytes.len()).context(\"FUSE callback probe is too large\")?;\n    stream.write_all(&length.to_be_bytes()).await?;\n    stream.write_all(&bytes).await?;\n    stream.flush().await?;\n    let response = read_callback_response(&mut stream).await?;\n    if response.request_id != request.request_id {\n        bail!(\"FUSE callback probe correlation mismatch\");\n    }\n    match response.outcome {\n        StorageFilesystemOutcomeV2::Success(_) => Ok(()),\n        StorageFilesystemOutcomeV2::Failure(StorageFilesystemFailureV1 { code, message }) => {\n            bail!(\"FUSE callback probe failed [{code}]: {message}\")\n        },\n    }\n}\n\nasync fn read_callback_response(stream: &mut LocalStream) -> Result<StorageFilesystemResponseV2> {\n    let mut length = [0_u8; 4];\n    stream.read_exact(&mut length).await?;\n    let length = u32::from_be_bytes(length) as usize;\n    if length == 0 || length > MAX_CALLBACK_BYTES {\n        bail!(\"FUSE callback response exceeds the bounded frame size\");\n    }\n    let mut bytes = vec![0_u8; length];\n    stream.read_exact(&mut bytes).await?;","sourceCodeStart":298,"sourceCodeEnd":334,"githubUrl":"https://github.com/astrid-runtime/astrid/blob/affd8760f44190dbdfbec23403f4c4b642c33112/crates/astrid-storage-provider-fuse/src/service.rs#L298-L334","documentation":"probe_callback sends a serialized request with a request_id over the local transport to the launch's callback socket, then reads a response that must carry the same request_id. If the response's request_id differs, the reply cannot be correlated with the probe (wrong peer, protocol desync, or a foreign service on the socket), and the launch probe fails.","triggerScenarios":"Calling run_launch when the callback endpoint answers with a response whose request_id field does not equal request.request_id — e.g. the socket is served by a different/older process, responses are interleaved from concurrent probes, or the peer mishandles the id field.","commonSituations":"A leftover service from a previous launch still listening on the callback path and answering with its own ids; concurrent launches sharing a callback socket causing crossed responses; a peer implementation bug echoing the wrong request_id; protocol version mismatch between kernel and provider.","solutions":["Verify the process listening on launch.lease.callback_path is the current launch's peer (kill stale instances from earlier launches)","Ensure the callback peer echoes request.request_id verbatim in its StorageFilesystemOutcomeV2 response","Serialize probes over a given callback socket — never issue concurrent probes that share one connection/path without multiplexing","Check kernel/provider version compatibility for the callback protocol framing"],"exampleFix":"// before: peer responds with a fixed id\nresponse.request_id = DEFAULT_ID;\n\n// after: echo the caller's id\nresponse.request_id = request.request_id;","handlingStrategy":"try-catch","validationCode":"if std::fs::read_dir(lease.resource_path.join(\"..\")).is_err() {\n    // lease directory gone/stale — relaunch before probing\n}","typeGuard":"fn is_probe_response(r: &CallbackResponse, want: u64) -> bool { r.request_id == want }","tryCatchPattern":"match probe_callback(launch).await {\n    Err(e) if e.to_string().contains(\"correlation mismatch\") => {\n        // stale/foreign peer on callback socket: kill old instance and relaunch\n    }\n    Err(e) => return Err(e),\n    Ok(()) => {}\n}","preventionTips":["Ensure exactly one peer owns each callback socket","Always echo request_id in callback responses","Avoid concurrent probes on a single callback socket","Align kernel/provider protocol versions"],"tags":["fuse","ipc","request-correlation","callback-probe","protocol"],"backgroundTag":"unexpected-response-shape","analyzedSha":"affd8760f44190dbdfbec23403f4c4b642c33112","analyzedAt":"2026-09-09T21:28:12.402Z","contentChangedAt":"2026-09-09T21:28:12.402Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}