{"record":{"id":"107d33a3ea1a90b2","repo":"block/buzz","slug":"system-message-insert-failed-e","errorCode":null,"errorMessage":"system message insert failed: {e}","messagePattern":"system message insert failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/buzz-relay/src/handlers/side_effects.rs","lineNumber":716,"sourceCode":"    channel_id: Uuid,\n    content: serde_json::Value,\n    idempotency_ts: chrono::DateTime<chrono::Utc>,\n) -> anyhow::Result<()> {\n    let channel_tag = Tag::parse([\"h\", &channel_id.to_string()])?;\n\n    let ts = nostr::Timestamp::from(idempotency_ts.timestamp() as u64);\n    let event = EventBuilder::new(Kind::Custom(40099), content.to_string())\n        .tags([channel_tag])\n        .custom_created_at(ts)\n        .sign_with_keys(&state.relay_keypair)\n        .map_err(|e| anyhow::anyhow!(\"failed to sign system message: {e}\"))?;\n\n    // Durable insert is the completion boundary — propagate failure.\n    state\n        .db\n        .insert_event(tenant.community(), &event, Some(channel_id))\n        .await\n        .map_err(|e| anyhow::anyhow!(\"system message insert failed: {e}\"))?;\n\n    // Fan out to subscribers: best-effort, clients can retrieve the persisted event.\n    if let Err(e) = state\n        .pubsub\n        .publish_event(tenant, EventTopic::Channel(channel_id), &event)\n        .await\n    {\n        warn!(\"System message fan-out failed: {e}\");\n    }\n\n    Ok(())\n}\n\n/// Sign and fan out a fresh relay-signed `kind:39005` thread-summary overlay\n/// for `root_id` after a thread mutation (reply insert or threaded delete).\n///\n/// Fan-out only — never stored. Channel-window pages recompute summaries from\n/// `thread_metadata` on every fetch (`api/bridge.rs`), so a persisted copy","sourceCodeStart":698,"sourceCodeEnd":734,"githubUrl":"https://github.com/block/buzz/blob/dad5a33865fc81a2e55b3b60746632f615ec1e3a/crates/buzz-relay/src/handlers/side_effects.rs#L698-L734","documentation":"This error is the completion boundary of `emit_system_message`: the relay-signed kind:40099 system event was built and signed, but the durable insert into the events store via `db.insert_event(tenant.community(), &event, Some(channel_id))` failed. The error is deliberately propagated (unlike the best-effort pubsub fan-out below it) because persistence of the system message is required for the triggering operation (group create, membership change, DM open, etc.) to be considered complete.","triggerScenarios":"Any emit_system_message caller when insert_event returns Err — Postgres down or timing out, unique-id conflict other than the ON CONFLICT DO NOTHING idempotency path, FK failure on channel_id (channel deleted concurrently), tenant/community row missing, or connection-pool exhaustion.","commonSituations":"DB failover or connection-pool exhaustion during a burst of membership operations; channel deleted between the membership event and the system-message insert causing FK violation; tenant/community misconfiguration making tenant.community() reference a nonexistent community; disk-full or replication lag on Postgres.","solutions":["Read the inner `{e}` to distinguish connectivity (retry) from constraint (fix data) causes.","Verify Postgres health and relay DB pool sizing; retry the triggering operation — the stable idempotency created_at makes retries produce the same event ID, so ON CONFLICT DO NOTHING dedupes.","If FK violation, check whether the channel was deleted concurrently and treat the operation as moot rather than retrying.","Confirm tenant.community() resolves to an existing community row for this tenant.","Check disk space / replication status on the Postgres host if inserts fail persistently."],"exampleFix":"// before: any insert error aborts the caller\nstate.db.insert_event(tenant.community(), &event, Some(channel_id)).await\n    .map_err(|e| anyhow::anyhow!(\"system message insert failed: {e}\"))?;\n// after: tolerate idempotent replay, propagate real failures\nmatch state.db.insert_event(tenant.community(), &event, Some(channel_id)).await {\n    Ok(()) => (),\n    Err(e) if is_duplicate_event(&e) => (), // already inserted by a prior attempt\n    Err(e) => return Err(anyhow::anyhow!(\"system message insert failed: {e}\")),\n}","handlingStrategy":"retry","validationCode":"// pre-flight: channel still exists and community resolves\nstate.db.get_channel(tenant.community(), channel_id).await\n    .map_err(|e| anyhow::anyhow!(\"pre-check channel {channel_id}: {e}\"))?;","typeGuard":"fn is_retryable_insert_error(e: &anyhow::Error) -> bool {\n    let s = format!(\"{e:#}\");\n    let transient = [\"connection\", \"timeout\", \"closed\", \"pool\"].iter().any(|k| s.contains(k));\n    let idempotent_dup = s.contains(\"duplicate\") && s.contains(\"events_pkey\");\n    transient || idempotent_dup\n}","tryCatchPattern":"match emit_system_message(&tenant, &state, channel_id, content, idempotency_ts).await {\n    Ok(()) => (),\n    Err(e) if is_retryable_insert_error(&e) => {\n        // stable idempotency_ts => same event id => ON CONFLICT DO NOTHING dedupes\n        backoff_retry(|| emit_system_message(&tenant, &state, channel_id, content, idempotency_ts), 3).await?\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Always pass a stable idempotency_ts (e.g. the outbox row's created_at) so retries dedupe via ON CONFLICT DO NOTHING.","Keep the same idempotency timestamp across retry attempts — a changing timestamp defeats event-id idempotency.","Monitor Postgres health, pool exhaustion, and disk space; alert on 'system message insert failed'.","Check channel existence before membership operations that emit system messages to avoid FK failures on concurrent deletes."],"tags":["database","postgresql","nostr","persistence","rust"],"backgroundTag":"event-insert-failed","analyzedSha":"dad5a33865fc81a2e55b3b60746632f615ec1e3a","analyzedAt":"2026-08-30T13:49:18.474Z","contentChangedAt":"2026-08-30T13:49:18.474Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}