stamparm/maltrail · error

condensed observable store: flush of

Error message

condensed observable store: flush of {drained} rows failed ({e})

What it means

This error is logged when the sensor's condensed observable store fails to persist a batch of drained in-memory observables (IPs and names) to its backing store during a periodic flush. The flush loop batches queued rows, calls write(), and on failure counts the batch as lost while logging the underlying error. Data in the failed batch was dropped from the queue and is not retried.

Solutions

1. Read the wrapped error {e} in the log message to identify the underlying store failure and fix it (disk space, permissions, connectivity).
2. Verify the condensed observable store path/credentials in the sensor configuration and restart the sensor.
3. Check host filesystem health (df, dmesg) if the store is file-backed; restore or recreate the store if corrupt.
4. Monitor flush_errors in sensor metrics; if it recurs, capture diagnostics before restarting.

// check store health before relying on flushes
// e.g. verify the backing store path/DB is writable and has free space
if !store_backend_reachable() { alert("condensed store unavailable; flushes will fail"); }

When it happens

Trigger: The internal _flush_loop calls flush() and self.write(&ips, &names) returns Err — e.g. the backing database/file is unavailable, full, or corrupt. Any rows drained into that batch are counted in flush_errors and lost.

Common situations: Disk full or permission problems on the host where the condensed store lives; the backing store process is down or unreachable; corrupted store file after an unclean shutdown; store write path misconfigured in sensor settings.

Related errors

• condensed observable store: prune failed

AI-assisted analysis of stamparm/maltrail@77cfb06d76 (2026-09-13). Data as JSON: /api/errors/e7af08c3ef129013. Report an issue: GitHub.

Appendix: source

Thrown at sensor/src/meta.rs:225

    /// Drain this worker's aggregate into SQLite. `core/meta.py:flush()`.
    ///
    /// A failure loses the window rather than retrying: the alternative is an unbounded in-RAM
    /// backlog on a host whose disk has gone read-only, which would turn a degraded auxiliary
    /// index into an OOM of the sensor itself. The error is logged, and the counter is exported.
    pub fn flush(&mut self) {
        if !self.enabled || self.pending() == 0 {
            return;
        }
        let ips = std::mem::take(&mut self.ips);
        let names = std::mem::take(&mut self.names);
        let drained = (ips.len() + names.len()) as u64;

        match self.write(&ips, &names) {
            Ok(()) => self.flushed += drained,
            Err(e) => {
                self.flush_errors += 1;
                crate::output::log_error(
                    &format!("condensed observable store: flush of {drained} rows failed ({e})"),
                    true,
                );
            }
        }
    }

    fn write(&self, ips: &FastMap<Ip, Row>, names: &StrMap<String, Row>) -> rusqlite::Result<()> {
        let mut con = open_rw(&self.db_path)?;
        // IMMEDIATE, not the default DEFERRED: this transaction only ever writes, and taking the
        // reserved lock up front is what lets `busy_timeout` do its job. A deferred transaction
        // that upgrades from read to write mid-way can be handed SQLITE_BUSY without the busy
        // handler being consulted at all, because SQLite cannot rule out a deadlock — which with
        // several workers draining into one file would show up as sporadic lost windows.
        let tx = con.transaction_with_behavior(rusqlite::TransactionBehavior::Immediate)?;
        {
            // Prepared once per flush and reused for every row; the pair below is the portable
            // merge core/meta.py uses (no ON CONFLICT, so any SQLite that can create the table

View on GitHub (pinned to 77cfb06d76)