astrid-runtime/astrid · error

FUSE callback probe failed

Error message

FUSE callback probe failed [{code}]: {message}

What it means

The FUSE filesystem callback probe round-trip completed, but the kernel-side callback reported a failure outcome instead of success. probe_callback() correlates the response by request_id and then bails with the failure's code and message, so the underlying remote failure text is surfaced here.

Solutions

1. Read the [{code}] and {message} embedded in the error — they come from the remote StorageFilesystemFailureV1 and identify the actual root cause.
2. Verify the admin/kernel service is running and healthy before launching the FUSE provider.
3. Check that the filesystem view/lease the probe refers to exists and the provider has access to it.
4. Re-run the launch (run_launch) after fixing the remote condition; the probe is a startup self-test and safe to retry.

Example fix

// before: startup aborts on any probe failure
probe_callback(&mut stream, request).await?;
// after: inspect the code and retry transient remote failures
match probe_callback(&mut stream, request).await {
    Ok(()) => {},
    Err(e) if e.to_string().contains("[temporary]") => {
        tokio::time::sleep(Duration::from_secs(1)).await;
        probe_callback(&mut stream, request).await?;
    },
    Err(e) => return Err(e),
}

// No pre-call check available; the failure comes from the remote callback handler.
// Optionally probe service health first if an admin status endpoint exists.

// Rust
match run_launch().await {
    Err(e) if e.to_string().starts_with("FUSE callback probe failed") => {
        // parse "[{code}]" and handle known transient codes with retry
    }
    other => other?,
}

When it happens

Trigger: Calling probe_callback() when the StorageFilesystemResponseV2 carries StorageFilesystemOutcomeV2::Failure(StorageFilesystemFailureV1 { code, message }) — i.e. the FUSE callback handler on the kernel/admin side rejected or failed the probe operation.

Common situations: The admin/kernel service is unhealthy or misconfigured when the provider starts and self-tests its callback channel; the callback endpoint lacks permissions or the view/mount referenced by the probe does not exist; a version mismatch causes the remote handler to fail the operation.

Related errors

• cannot unmount a relative mountpoint
• detached FUSE service access
• detached FUSE service did not retain its control endpoint
• detached FUSE service exceeded the startup response size
• detached FUSE service failed

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/2b6a4903e465ee5b. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-storage-provider-fuse/src/service.rs:321

        request_id: format!("fuse-service-{}", Uuid::new_v4()),
        lease_token: launch.lease.lease_token.clone(),
        operation: StorageFilesystemOperationV2::Stat {
            path: String::new(),
        },
    };
    let bytes = serde_json::to_vec(&request).context("encode FUSE callback probe")?;
    let length = u32::try_from(bytes.len()).context("FUSE callback probe is too large")?;
    stream.write_all(&length.to_be_bytes()).await?;
    stream.write_all(&bytes).await?;
    stream.flush().await?;
    let response = read_callback_response(&mut stream).await?;
    if response.request_id != request.request_id {
        bail!("FUSE callback probe correlation mismatch");
    }
    match response.outcome {
        StorageFilesystemOutcomeV2::Success(_) => Ok(()),
        StorageFilesystemOutcomeV2::Failure(StorageFilesystemFailureV1 { code, message }) => {
            bail!("FUSE callback probe failed [{code}]: {message}")
        },
    }
}

async fn read_callback_response(stream: &mut LocalStream) -> Result<StorageFilesystemResponseV2> {
    let mut length = [0_u8; 4];
    stream.read_exact(&mut length).await?;
    let length = u32::from_be_bytes(length) as usize;
    if length == 0 || length > MAX_CALLBACK_BYTES {
        bail!("FUSE callback response exceeds the bounded frame size");
    }
    let mut bytes = vec![0_u8; length];
    stream.read_exact(&mut bytes).await?;
    let response: StorageFilesystemResponseV2 =
        serde_json::from_slice(&bytes).context("decode FUSE callback probe")?;
    if response.protocol_version != STORAGE_FILESYSTEM_PROTOCOL_V2 {
        bail!("FUSE callback probe protocol mismatch");
    }

View on GitHub (pinned to affd8760f4)