projection order drift should be ignored even with apostroph

Error message

projection order drift should be ignored even with apostrophes in comments

What it means

Test panic from .expect at crates/dbt-adapter/src/sql/diff.rs:3633: compare_sql(actual, expected, AdapterType::Bigquery) returned Err for queries whose only difference is projection order plus SQL comments containing apostrophes. The tokenizer/canonicalizer must strip comments before matching projections; failing here means comment handling (specifically apostrophe/quote characters inside /* */ comments) corrupts tokenization so the projection reordering never applies.

Source

Thrown at crates/dbt-adapter/src/sql/diff.rs:3633

        let expected_sql = format!(
            "select {} from t",
            expected_alias_order
                .iter()
                .map(|a| projection_for(a))
                .collect::<Vec<_>>()
                .join(", ")
        );
        let actual_sql = format!(
            "select {} from t",
            actual_alias_order
                .iter()
                .map(|a| projection_for(a))
                .collect::<Vec<_>>()
                .join(", ")
        );

        compare_sql(&actual_sql, &expected_sql, AdapterType::Bigquery)
            .expect("projection order drift should be ignored even with apostrophes in comments");
    }

    #[test]
    fn test_bigquery_simple_projection_order_drift_with_comment_apostrophe_should_be_ignorable() {
        // Minimal repro for the next Snowplow ordering drift surface:
        //
        // Downstream models may select the already-built context columns as plain identifiers:
        //   , experiment_entity
        //   , feature_flag_context
        //   , ...
        //
        // Mantle vs Fusion can emit these in different orders due to map iteration. This should be
        // ignorable for replay, even when the SELECT list contains `--` comments with apostrophes.
        //
        // This test is expected to FAIL (SqlMismatch) until we canonicalize identifier projection
        // ordering drift for this pattern.
        let expected_order = [
            "service_configuration_context",

View on GitHub (pinned to 0267ce9170)

Solutions

Strip /* */ and -- comments (respecting string literals) before projection matching, without treating apostrophes inside comments as string delimiters
Inspect the tokenizer's quote-state machine: ensure comment state consumes characters until */ regardless of embedded quotes
Compare with the passing no-comment test (diff.rs:3559) to confirm the delta is purely comment handling
Add a unit test for the tokenizer with `/* it's */` style comments

Example fix

// before: tokenizer treats ' inside /* */ as string start
let sql = "select a /* user's col */, b";
// after: comment scanner consumes until */ ignoring quotes
let stripped = strip_comments(sql); // -> "select a , b"
compare_sql(stripped_actual, stripped_expected, AdapterType::Bigquery)
    .expect("projection order drift should be ignored even with apostrophes in comments");

Defensive patterns

Strategy: validation

Validate before calling

// strip comments safely before any comparison
fn strip_comments(sql: &str) -> String { /* consume -- and /* */ respecting string literals */ }
assert_eq!(strip_comments("select a /* it's */"), "select a");

Try / catch

let (a, b) = (strip_comments(&actual), strip_comments(&expected));
compare_sql(&a, &b, AdapterType::Bigquery)
    .unwrap_or_else(|e| eprintln!("comment-stripped diff still fails: {e}"));

Prevention

Strip comments before tokenization in any SQL comparison pipeline
Make the comment scanner ignore quote characters inside /* */
Add apostrophe-in-comment fixtures to tokenizer tests
Keep string-literal detection and comment detection in one state machine

When it happens

Trigger: cargo test -p dbt-adapter with test at diff.rs:3633: the minimal Snowplow repro whose comments contain apostrophes (e.g. `/* user's metric */`) makes compare_sql under AdapterType::Bigquery report a mismatch that the same test without apostrophes (diff.rs:3559) does not.

Common situations: Comments with single quotes in hand-written or generated SQL confuse the string-literal scanner, causing apostrophe-balancing bugs that shift token boundaries and break alias extraction; replay of Snowplow-style models with commented projections fails only on comment-bearing lines.

Related errors

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/51f5c5ecb4074994. Report an issue: GitHub.