block/buzz · error

system message insert failed: {e}

Error message

system message insert failed: {e}

What it means

This error is the completion boundary of `emit_system_message`: the relay-signed kind:40099 system event was built and signed, but the durable insert into the events store via `db.insert_event(tenant.community(), &event, Some(channel_id))` failed. The error is deliberately propagated (unlike the best-effort pubsub fan-out below it) because persistence of the system message is required for the triggering operation (group create, membership change, DM open, etc.) to be considered complete.

Source

Thrown at crates/buzz-relay/src/handlers/side_effects.rs:716

    channel_id: Uuid,
    content: serde_json::Value,
    idempotency_ts: chrono::DateTime<chrono::Utc>,
) -> anyhow::Result<()> {
    let channel_tag = Tag::parse(["h", &channel_id.to_string()])?;

    let ts = nostr::Timestamp::from(idempotency_ts.timestamp() as u64);
    let event = EventBuilder::new(Kind::Custom(40099), content.to_string())
        .tags([channel_tag])
        .custom_created_at(ts)
        .sign_with_keys(&state.relay_keypair)
        .map_err(|e| anyhow::anyhow!("failed to sign system message: {e}"))?;

    // Durable insert is the completion boundary — propagate failure.
    state
        .db
        .insert_event(tenant.community(), &event, Some(channel_id))
        .await
        .map_err(|e| anyhow::anyhow!("system message insert failed: {e}"))?;

    // Fan out to subscribers: best-effort, clients can retrieve the persisted event.
    if let Err(e) = state
        .pubsub
        .publish_event(tenant, EventTopic::Channel(channel_id), &event)
        .await
    {
        warn!("System message fan-out failed: {e}");
    }

    Ok(())
}

/// Sign and fan out a fresh relay-signed `kind:39005` thread-summary overlay
/// for `root_id` after a thread mutation (reply insert or threaded delete).
///
/// Fan-out only — never stored. Channel-window pages recompute summaries from
/// `thread_metadata` on every fetch (`api/bridge.rs`), so a persisted copy

View on GitHub (pinned to dad5a33865)

Solutions

1. Read the inner `{e}` to distinguish connectivity (retry) from constraint (fix data) causes.
2. Verify Postgres health and relay DB pool sizing; retry the triggering operation — the stable idempotency created_at makes retries produce the same event ID, so ON CONFLICT DO NOTHING dedupes.
3. If FK violation, check whether the channel was deleted concurrently and treat the operation as moot rather than retrying.
4. Confirm tenant.community() resolves to an existing community row for this tenant.
5. Check disk space / replication status on the Postgres host if inserts fail persistently.

Example fix

// before: any insert error aborts the caller
state.db.insert_event(tenant.community(), &event, Some(channel_id)).await
    .map_err(|e| anyhow::anyhow!("system message insert failed: {e}"))?;
// after: tolerate idempotent replay, propagate real failures
match state.db.insert_event(tenant.community(), &event, Some(channel_id)).await {
    Ok(()) => (),
    Err(e) if is_duplicate_event(&e) => (), // already inserted by a prior attempt
    Err(e) => return Err(anyhow::anyhow!("system message insert failed: {e}")),
}

// pre-flight: channel still exists and community resolves
state.db.get_channel(tenant.community(), channel_id).await
    .map_err(|e| anyhow::anyhow!("pre-check channel {channel_id}: {e}"))?;

fn is_retryable_insert_error(e: &anyhow::Error) -> bool {
    let s = format!("{e:#}");
    let transient = ["connection", "timeout", "closed", "pool"].iter().any(|k| s.contains(k));
    let idempotent_dup = s.contains("duplicate") && s.contains("events_pkey");
    transient || idempotent_dup
}

match emit_system_message(&tenant, &state, channel_id, content, idempotency_ts).await {
    Ok(()) => (),
    Err(e) if is_retryable_insert_error(&e) => {
        // stable idempotency_ts => same event id => ON CONFLICT DO NOTHING dedupes
        backoff_retry(|| emit_system_message(&tenant, &state, channel_id, content, idempotency_ts), 3).await?
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Any emit_system_message caller when insert_event returns Err — Postgres down or timing out, unique-id conflict other than the ON CONFLICT DO NOTHING idempotency path, FK failure on channel_id (channel deleted concurrently), tenant/community row missing, or connection-pool exhaustion.

Common situations: DB failover or connection-pool exhaustion during a burst of membership operations; channel deleted between the membership event and the system-message insert causing FK violation; tenant/community misconfiguration making tenant.community() reference a nonexistent community; disk-full or replication lag on Postgres.

Related errors

• delete failed: {e}
• channel not found
• moderation notice recipient must be a 32-byte pubkey, got {}
• failed to sign moderation notice: {e}
• ban failed: {e}

AI-assisted analysis of block/buzz@dad5a33865 (2026-08-30). Data as JSON: /api/errors/107d33a3ea1a90b2. Report an issue: GitHub.