risingwavelabs/risingwave · critical · MetaError

Integrity check failed

Error message

Integrity check failed

What it means

MetaError::IntegrityCheckFailed indicates that a metadata integrity verification failed on the meta node. The catalog/integrity checker validates invariants between internal metadata models (e.g. streaming job fragments, tables, sink references); when an invariant is broken (dangling references, inconsistent states), this error is produced so operators know the metadata store is inconsistent and may need manual repair.

Solutions

1. Read the integrity checker logs to identify which invariant and objects failed
2. Back up the metadata database before any manual repair
3. Remove/repair the orphaned or inconsistent metadata entries identified by the checker
4. Re-run the integrity check to confirm the metadata is clean
5. Report to RisingWave maintainers with logs if this arises without manual intervention

// Run the built-in integrity check proactively after crash/upgrade
// (meta service routine)
if let Err(e) = meta_service.run_integrity_check().await {
    tracing::error!("metadata inconsistent: {e}");
    // alert and take meta store backup before repair
}

fn is_integrity_failure(e: &MetaError) -> bool {
    matches!(e, MetaError::IntegrityCheckFailed)
}

When it happens

Trigger: Running the metadata integrity check routine in the meta service (`catalog::integrity_check` / verify_metadata_invariants) on a cluster whose metadata has dangling objects, orphaned fragments, or partially-created jobs (e.g. after a crash mid-DDL or a manual meta store edit).

Common situations: Crash during DDL (table/mv/sink creation or deletion) leaving orphan records; manual edits to the metadata database; version-upgrade migrations leaving stale entries; bugs in concurrent DDL handling.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• has been deprecated, please use instead.
• adhoc recovery triggered
• Can't create iceberg sink write result from empty data!
• Can't find schema by id
• Cannot alter the job

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/903fccdc65375de9. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/error.rs:135

    #[error("Sink error: {0}")]
    Sink(
        #[from]
        #[backtrace]
        SinkError,
    ),

    #[error(transparent)]
    Internal(
        #[from]
        #[backtrace]
        anyhow::Error,
    ),

    // Indicates that recovery was triggered manually.
    #[error("adhoc recovery triggered")]
    AdhocRecovery,

    #[error("Integrity check failed")]
    IntegrityCheckFailed,

    #[error("{0} has been deprecated, please use {1} instead.")]
    Deprecated(String, String),

    #[error(transparent)]
    NotImplemented(#[from] NotImplemented),

    #[error("Secret error: {0}")]
    SecretError(
        #[from]
        #[backtrace]
        SecretError,
    ),
}

impl MetaError {
    /// Provide the Postgres error code for the error.

View on GitHub (pinned to 6469eb736d)