What causes "should split on top-level UNION" in dbt-labs/dbt-core?

Iterate chars (char_indices) instead of raw byte indexing so multi-byte characters cannot corrupt the scan position

should split on top-level UNION

Error message

should split on top-level UNION

What it means

Test panic from Option::expect at crates/dbt-adapter/src/sql/diff.rs:3768: split_union_top_level(sql) returned None, meaning the scanner failed to find a top-level UNION keyword outside parentheses/strings/comments, so it could not return the vector of statement parts. The splitter must skip multi-byte characters safely (the regression this test guards: an index landing mid-UTF-8 char in “unicode” could panic or miss the UNION).

Source

Thrown at crates/dbt-adapter/src/sql/diff.rs:3768

    partition by billing_group_id
    order by __as_of
    rows between unbounded preceding and current row
  ) as a,
  rn
from filled_data
qualify row_number() over (partition by billing_group_id, __as_of order by rn desc) = 1
"#;

        compare_sql(sql_fusion, sql_recorded, AdapterType::Snowflake)
            .expect("Forward-fill projection column order drift should be ignored");
    }

    #[test]
    fn test_split_union_top_level_splits_and_handles_unicode() {
        // Regression test: previously this could panic if the scan index landed in the middle
        // of a multi-byte UTF-8 char (e.g. “).
        let sql = "select 1 as a /* “unicode” */ UNION      select 2 as b";
        let parts = split_union_top_level(sql).expect("should split on top-level UNION");
        assert_eq!(parts, vec!["select 1 as a", "select 2 as b"]);
    }

    #[test]
    fn test_split_union_all_top_level_splits_and_handles_unicode() {
        // Regression test: previously this could panic if the scan index landed in the middle
        // of a multi-byte UTF-8 char (e.g. “).
        let sql = "select 1 as a /* “unicode” */ UNION   ALL   select 2 as b";
        let parts = split_union_all_top_level(sql).expect("should split on top-level UNION ALL");
        assert_eq!(parts, vec!["select 1 as a", "select 2 as b"]);
    }

    #[test]
    fn test_split_union_all_top_level_does_not_split_inside_parentheses() {
        let sql = "select 1 as a union all select (select 2 as b union all select 3 as c)";
        let parts =
            split_union_all_top_level(sql).expect("should split on the top-level UNION ALL");
        assert_eq!(

View on GitHub (pinned to 0267ce9170)

Solutions

Iterate chars (char_indices) instead of raw byte indexing so multi-byte characters cannot corrupt the scan position
Verify the comment scanner consumes /* “unicode” */ entirely before looking for UNION at top level
Normalize whitespace around the UNION keyword before matching, and ensure case-insensitive matching
If the SQL genuinely has no top-level UNION, confirm returning None is intended and the test input is correct

Example fix

// before (byte indexing can split a multi-byte char)
while i < s.len() { let b = s.as_bytes()[i]; ... }
// after
for (i, c) in s.char_indices() { /* scan, skip comments/strings, detect top-level UNION */ }

Defensive patterns

Strategy: try-catch

Validate before calling

// confirm a top-level UNION exists before calling the splitter
fn has_top_level_union(sql: &str) -> bool {
    scan_top_level(sql, |s| s.eq_ignore_ascii_case("UNION"))
}
if !has_top_level_union(sql) { /* skip split path */ }

Type guard

fn is_split(parts: &Option<Vec<String>>) -> bool { matches!(parts, Some(v) if v.len() > 1) }

Try / catch

match split_union_top_level(sql) {
    Some(parts) => compare_parts(parts),
    None => compare_whole(sql), // graceful fallback instead of expect
}

Prevention

Use char_indices, never byte indexing, when scanning SQL text
Skip comments/strings before keyword matching
Whitespace-tolerant keyword matching for set operators
Return None deliberately and handle it, rather than expect()ing in callers

When it happens

Trigger: cargo test -p dbt-adapter with test_split_union_top_level_splits_and_handles_unicode: input "select 1 as a /* “unicode” */ UNION select 2 as b" and split_union_top_level returns None, panicking "should split on top-level UNION".

Common situations: SQL containing smart quotes or other multi-byte characters in comments near set operators; byte-index scanning after a char-boundary fix broke keyword detection; callers in compare_sql_structurally (diff.rs:1418) pass comment-bearing UNION queries that previously split fine.

Related errors

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/4dbee99a9d398ed0. Report an issue: GitHub.