BoundaryML/baml · error

BAML_CACHE_VERIFY: cached `test --list` testset discovery…

Error message

BAML_CACHE_VERIFY: cached `test --list` testset discovery differs from a fresh discovery ({} vs {} leaf tests). A testset generator is nondeterministic or reads uncached state (IO/env/LLM) — please report this.

What it means

Error "BAML_CACHE_VERIFY: cached `test --list` testset discovery differs from a fresh discovery ({} vs {} leaf tests). A testset generator is nondeterministic or reads uncached state (IO/env/LLM) — please report this." thrown in BoundaryML/baml.

Solutions

  1. Find the nondeterministic testset generator in your .baml sources and remove hidden state reads (file IO, environment variables, clock, or LLM calls) from generator code
  2. Clear the test discovery cache and regenerate it once the generator is deterministic
  3. If the generator looks clean, report the issue to the BAML maintainers with a minimal reproducible testset — a compile/discovery input may be missing from the cache key
Defensive patterns

Strategy: try-catch

When it happens

Trigger: Thrown at baml_language/crates/baml_cli/src/bytecode_cache.rs:494 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of BoundaryML/baml@bd85ce9dee (2026-09-12). Data as JSON: /api/errors/c94418cddddcc51a. Report an issue: GitHub.

Appendix: source

Thrown at baml_language/crates/baml_cli/src/bytecode_cache.rs:494

        // Read the on-disk blob directly, bypassing the `BAML_NO_DISCOVERY_CACHE`
        // disable (verify must still compare against whatever is on disk).
        let Some(cached) =
            self.load_decoded::<TestDiscovery>(&self.test_discovery_key, "test discovery")
        else {
            return Ok(());
        };
        Self::compare_test_discovery(&cached, honest)
    }

    /// The env-independent core of [`Self::verify_test_discovery`], so the
    /// oracle's discriminating power (pass on a faithful cache, bail on a stale
    /// one) is unit-testable without mutating the process environment.
    pub(crate) fn compare_test_discovery(
        cached: &TestDiscovery,
        honest: &TestDiscovery,
    ) -> anyhow::Result<()> {
        if cached.testset_leaf_names != honest.testset_leaf_names {
            anyhow::bail!(
                "BAML_CACHE_VERIFY: cached `test --list` testset discovery differs from a fresh \
                 discovery ({} vs {} leaf tests). A testset generator is nondeterministic or reads \
                 uncached state (IO/env/LLM) — please report this.",
                cached.testset_leaf_names.len(),
                honest.testset_leaf_names.len(),
            );
        }
        Ok(())
    }
}

/// Derive and borsh-serialize each stdlib package's `PackageInterface` from a
/// compiled database, keyed by package name. On a warm database the query
/// returns the seed verbatim, so re-serializing reproduces the same bytes
/// (idempotent); on a cold database it materializes the interface once.
fn extract_stdlib_interface(db: &ProjectDatabase) -> std::collections::BTreeMap<String, Vec<u8>> {
    use baml_db::baml_compiler2_hir_ty::package_interface::export_interface;
    let mut out = std::collections::BTreeMap::new();

View on GitHub (pinned to bd85ce9dee)