{"record":{"id":"33f202d139bfc2f1","repo":"risingwavelabs/risingwave","slug":"database-unavailable","errorCode":null,"errorMessage":"database {} unavailable {}","messagePattern":"database (.+?) unavailable (.+?)","errorType":"error_code","errorClass":"MetaError","httpStatus":null,"severity":"error","filePath":"src/meta/src/barrier/schedule.rs","lineNumber":610,"sourceCode":"\n    /// Mark command scheduler as blocked and abort all queued scheduled command and notify with\n    /// specific reason.\n    pub(super) fn abort_and_mark_blocked(\n        &self,\n        database_id: Option<DatabaseId>,\n        reason: impl Into<String>,\n    ) {\n        let mut queue = self.inner.queue.lock();\n        fn database_blocked_reason(database_id: DatabaseId, reason: &String) -> String {\n            format!(\"database {} unavailable {}\", database_id, reason)\n        }\n        fn mark_blocked_and_notify_failed(\n            database_id: DatabaseId,\n            queue: &mut DatabaseScheduledQueue,\n            reason: &String,\n        ) {\n            let reason = database_blocked_reason(database_id, reason);\n            let err: MetaError = anyhow!(\"{}\", reason).into();\n            queue.mark_blocked(reason);\n            while let Some(ScheduledQueueItem { notifier, .. }) = queue.queue.pop_front() {\n                notifier.notify_start_failed(err.clone());\n            }\n        }\n        if let Some(database_id) = database_id {\n            let reason = reason.into();\n            match queue.queue.entry(database_id) {\n                Entry::Occupied(entry) => {\n                    let queue = entry.into_mut();\n                    if queue.status.is_blocked() {\n                        if cfg!(debug_assertions) {\n                            panic!(\"database {} marked as blocked twice\", database_id);\n                        } else {\n                            warn!(?database_id, \"database marked as blocked twice\");\n                        }\n                    }\n                    info!(?database_id, \"database marked as blocked\");","sourceCodeStart":592,"sourceCodeEnd":628,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/barrier/schedule.rs#L592-L628","documentation":"mark_blocked_and_notify_failed marks a database's barrier queue as blocked and fails every pending queued start with a MetaError whose text is the database-blocked reason (rendered as `database <id> unavailable <reason>`). It is invoked when aborting barriers after detecting the database must stop making progress (e.g. it was dropped, paused, or its scheduling was aborted).","triggerScenarios":"abort_and_mark_blocked is called for a database id and the queue has pending items: the database was dropped/paused, an admin command blocked it, or barrier scheduling for that database was aborted; every waiting `notify_start_failed` sender receives this cloned error.","commonSituations":"A user dropped or paused a database while streaming jobs/barriers were in flight; an operator aborted a stuck barrier schedule; automated tooling issued aborts during failover or migration.","solutions":["Read the `<reason>` suffix in the message — it states why the database was blocked (dropped, paused, aborted, etc.).","If the database was dropped intentionally, stop clients targeting it and recreate it if needed.","If the block was unintentional, re-enable/unblock the database (admin command or recovery) and retry the operation.","During failover, wait for recovery to complete and reconnect before issuing new barrier-dependent queries."],"exampleFix":"// before: retrying blindly after database was blocked\nloop { run_query(); }\n// after: check database state before retrying\nif let Err(e) = run_query().await {\n    if is_database_blocked(&e) { unblock_or_recreate_database(db_id).await?; }\n}","handlingStrategy":"validation","validationCode":"-- ensure the database exists and is not blocked/paused before issuing work\nSELECT id, state FROM rw_databases WHERE id = <database_id>;\n-- abort if state indicates dropped/paused","typeGuard":null,"tryCatchPattern":"// treat as terminal for this database; do not blind-retry\nif is_database_blocked(&err) {\n    stop_pending_work_for(db_id);\n    // re-enable or recreate the database, then resubmit\n}","preventionTips":["Coordinate DROP/PAUSE DATABASE with running clients and streaming jobs.","Drain in-flight work before aborting barrier schedules.","Subscribe to database state changes so clients stop early instead of failing on the queue."],"tags":["meta","database","barrier","blocked-database"],"backgroundTag":"database-blocked","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}