block/buzz · critical

git conformance probe failed: {e}

Error message

git conformance probe failed: {e}

What it means

A fatal startup gate in buzz-relay: before serving git traffic, main() runs run_conformance_probe (crates/buzz-relay/src/api/git/store.rs:576) against the configured S3/MinIO backend (GitStore is built from the same BUZZ_S3_* media config, state.rs:836) to verify the linearizable conditional-write axiom A3. Four phases execute: sequential write/read-back, race_width parallel If-Match CAS updates of one pointer (exactly one winner allowed), race_width parallel create-only If-None-Match writes, and an ETag read-to-CAS round-trip. Any phase failure returns StoreError::Probe(ProbeFailure{phase, round, key, reason}); transport/auth problems return StoreError::Backend. main.rs wraps either as this error and the relay refuses to come up, because a backend without linearizable pointer CAS silently corrupts the git manifest-pointer protocol.

Source

Thrown at crates/buzz-relay/src/main.rs:528

            .unwrap_or(32);
        let race_rounds = std::env::var("BUZZ_GIT_PROBE_ROUNDS")
            .ok()
            .and_then(|v| v.parse().ok())
            .unwrap_or(3);
        let cfg = buzz_relay::api::git::store::ProbeConfig {
            race_width,
            race_rounds,
        };
        tracing::info!(
            race_width,
            race_rounds,
            "running git object-store conformance probe (A3 gate)"
        );
        let report = state
            .git_store
            .run_conformance_probe(cfg)
            .await
            .map_err(|e| anyhow::anyhow!("git conformance probe failed: {e}"))?;
        tracing::info!(
            race_width = report.race_width,
            race_rounds = report.race_rounds,
            transport_drops = report.transport_drops,
            "git object-store backend admitted: A3 conformance probe passed"
        );
    }

    // NIP-43: reconcile the event-backed roster for every provisioned
    // community before opening the listener. `relay_members` is canonical;
    // this repairs pre-snapshot communities and any publication that failed
    // after a membership transaction committed.
    if config.require_relay_membership {
        match buzz_relay::handlers::side_effects::reconcile_nip43_membership_snapshots(&state).await
        {
            Ok(count) => info!(count, "NIP-43 membership snapshots reconciled on startup"),
            Err(error) => {
                tracing::warn!(%error, "NIP-43 membership snapshot startup reconciliation failed")

View on GitHub (pinned to f956e6fe06)

Solutions

1. Read the phase and reason inside the error text: phase=sequential points at credentials/read-after-write, phase=if_match_race/if_none_match_race at non-linearizable conditional writes, phase=etag_consistency at unstable ETag tokens, phase=config at bad BUZZ_GIT_PROBE_* values.
2. Reproduce against the backend directly with the store's live probe: BUZZ_GIT_S3_PROBE=1 cargo test -p buzz-relay --lib using your BUZZ_S3_* values (see store.rs test docs).
3. Fix the backend: use an S3 implementation with real conditional writes and stable ETags (AWS S3, current MinIO), and correct BUZZ_S3_ENDPOINT/BUZZ_S3_BUCKET/keys/region and BUZZ_S3_ADDRESSING_STYLE.
4. If transport flakiness caused racer drops to fail a round, re-run with a smaller BUZZ_GIT_PROBE_WRITERS (default 32) — it must stay >= 2 and BUZZ_GIT_PROBE_ROUNDS >= 1.
5. Non-production only: set BUZZ_GIT_CONFORMANCE_PROBE=false to skip the gate, accepting an unsafe backend for git traffic.

Example fix

# before (relay exits: "git conformance probe failed: probe phase=if_none_match_race ... multiple winners")
BUZZ_S3_ENDPOINT=http://minio:9000
BUZZ_S3_BUCKET=buzz-media

# after — either fix the backend (upgrade MinIO to a conditional-write build),
# or explicitly skip the gate in dev only:
BUZZ_GIT_CONFORMANCE_PROBE=false

// Before boot, canary-check the S3 backend with a cheap probe run
// (minimum legal width, 1 round) so deploy pipelines fail before the relay does:
let cfg = ProbeConfig { race_width: 2, race_rounds: 1 };
let report = git_store.run_conformance_probe(cfg).await
    .map_err(|e| anyhow::anyhow!("deploy blocked: S3 backend not conformant: {e}"))?;
if report.transport_drops > 0 {
    tracing::warn!(drops = report.transport_drops, "S3 endpoint is flaky");
}

use buzz_relay::api::git::store::StoreError;

// Backend answered but violated an axiom (replace/upgrade the backend).
fn is_conformance_violation(e: &StoreError) -> bool {
    matches!(e, StoreError::Probe(_))
}

// Backend never answered usefully (creds/endpoint/transport).
fn is_backend_reachability(e: &StoreError) -> bool {
    matches!(e, StoreError::Backend(_))
}

match state.git_store.run_conformance_probe(cfg).await {
    Ok(report) => tracing::info!(?report, "backend admitted"),
    Err(e) if matches!(e, StoreError::Probe(_)) => {
        return Err(anyhow!("backend reachable but non-conformant — replace it: {e}"))
    }
    Err(e) => {
        return Err(anyhow!("backend unreachable/misconfigured — check BUZZ_S3_*: {e}"))
    }
}

When it happens

Trigger: Booting with BUZZ_GIT_CONFORMANCE_PROBE unset or != "false" (probe runs by default) while the BUZZ_S3_* backend is unreachable, has bad credentials/bucket, or is S3-compatible but not conditional-write-conformant. Specific probe failures: 'sequential' read-after-write mismatch or 403 on PUT; 'if_match_race'/'if_none_match_race' with more than one 2xx winner (backend ignores If-Match/If-None-Match or returns 200 instead of 412); 'etag_consistency' when the ETag changes between get and IfMatch; missing ETag header on a successful CAS; 'config' phase when BUZZ_GIT_PROBE_WRITERS < 2 or BUZZ_GIT_PROBE_ROUNDS = 0.

Common situations: MinIO or an S3 gateway that lacks or weakens conditional writes / rewrites ETags (older MinIO, some object-store proxies); wrong credentials so every PUT fails; CI spinning up a MinIO without conditional-write support; pointing BUZZ_S3_ENDPOINT at the wrong scheme or port; developers who only want chat features and hit an unexpected S3 gate at boot.

Related errors

• invalid media config: {e}
• failed to initialize media storage: {e}
• Failed to bind health port {}: {e}
• Failed to bind {}: {e}
• BUZZ_UDS_PATH {uds_path} exists but is not a socket

AI-assisted analysis of block/buzz@f956e6fe06 (2026-08-16). Data as JSON: /api/errors/038623e20a78fc91. Report an issue: GitHub.