dbt-labs/dbt-core · error

reduce key originates from relations_by_schema

Error message

reduce key originates from relations_by_schema

What it means

In `run_via_show_tables` (crates/dbt-adapter/src/metadata/redshift/mod.rs:860), the reduce closure re-fetches `relations_by_schema.get(&key)` and asserts the key is always present because keys were produced from `relations_by_schema.keys()`. The panic fires only if the map was mutated/cleared between building the keys and running the map-reduce, or the key set was derived inconsistently.

Solutions

1. Ensure `keys` is derived from the exact same `relations_by_schema` instance passed to the reduce closure and is not mutated during the run
2. Replace `.expect` with graceful handling: skip unknown keys and return Ok
3. Clone the relations entry into the key tuple so no lookup is needed in reduce
4. Add a debug assertion/log capturing the missing key to catch the producer bug

Example fix

// before
.expect("reduce key originates from relations_by_schema");
// after
let Some(relations) = relations_by_schema.get(&key) else { return Ok(()) };

// ensure keys and map come from the same source
assert!(keys.iter().all(|k| relations_by_schema.contains_key(k)));

fn keys_match_map(keys: &[Key], map: &HashMap<Key, Relations>) -> bool {
    keys.iter().all(|k| map.contains_key(k))
}

std::panic::catch_unwind(AssertUnwindSafe(|| map_reduce.run(Arc::new(keys), token)))

When it happens

Trigger: Running the Redshift SHOW TABLES freshness map-reduce after `relations_by_schema` has been modified or a key was produced from a different map instance (e.g. cloned relations map while iterating keys of another).

Common situations: Concurrent mutation of the relations map during the run; refactors that build `keys` from a filtered/different collection; partial map-reduce retries with stale keys.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Adapter must be configured for the parse phase
• Adapter should be available during parse phase
• agate_table
• AgateTable exists
• All snapshot macros should start with 'snapshot_'

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/3b1744b9af246410. Report an issue: GitHub.

Appendix: source

Thrown at crates/dbt-adapter/src/metadata/redshift/mod.rs:860

                quote_ident(AdapterType::Redshift, database),
                quote_ident(AdapterType::Redshift, schema),
            );
            let ctx = QueryCtx::default().with_desc("Extracting freshness via SHOW TABLES");
            let (_, agate_table) =
                adapter.query(&ctx, &mut *conn, &sql, None, token_clone.clone())?;
            Ok(agate_table.original_record_batch())
        };

        type Acc = BTreeMap<String, MetadataFreshness>;
        let reduce_f = move |acc: &mut Acc,
                             key: (String, String),
                             batch_res: AdapterResult<Arc<RecordBatch>>|
              -> Result<(), Cancellable<AdapterError>> {
            let batch = batch_res?;
            // `key` came from `relations_by_schema.keys()`, so it is always present.
            let relations = relations_by_schema
                .get(&key)
                .expect("reduce key originates from relations_by_schema");
            acc.extend(parse_show_tables_freshness_batch(&batch, relations)?);
            Ok(())
        };

        let map_reduce = MapReduce::new(factory, Box::new(map_f), Box::new(reduce_f), None);
        map_reduce.run(Arc::new(keys), token)
    }
}

/// Parse a `SHOW TABLES FROM SCHEMA` result batch into a freshness map keyed by
/// each matching relation's semantic FQN. Rows whose table is not in the
/// requested set are ignored.
fn parse_show_tables_freshness_batch(
    batch: &RecordBatch,
    relations: &[Arc<dyn BaseRelation>],
) -> AdapterResult<BTreeMap<String, MetadataFreshness>> {
    let mut out = BTreeMap::new();
    if batch.num_rows() == 0 {

View on GitHub (pinned to 0267ce9170)