vxcontrol/pentagi · error

failed to load test registry: %w

Error message

failed to load test registry: %w

What it means

TestProvider wraps any error from testdata.LoadBuiltinRegistry() with this message. The builtin registry is the embedded set of test suites (prompts, tool-use scenarios, etc.) that the provider tester runs; if it cannot be loaded/parsed from the embedded testdata, no tests can be run and the whole call aborts.

Source

Thrown at backend/pkg/providers/tester/runner.go:46

	agentType pconfig.ProviderOptionsType
	result    testdata.TestResult
	err       error
}

// TestProvider executes tests for a provider with given options
func TestProvider(ctx context.Context, prv provider.Provider, opts ...TestOption) (ProviderTestResults, error) {
	config := applyOptions(opts)

	// load test registry
	var registry *testdata.TestRegistry
	var err error

	if config.customRegistry != nil {
		registry = config.customRegistry
	} else {
		registry, err = testdata.LoadBuiltinRegistry()
		if err != nil {
			return ProviderTestResults{}, fmt.Errorf("failed to load test registry: %w", err)
		}
	}

	// collect all test requests
	requests := collectTestRequests(registry, prv, config)
	if len(requests) == 0 {
		return ProviderTestResults{}, fmt.Errorf("no tests to execute")
	}

	// execute tests in parallel
	responses := executeTestsParallel(ctx, requests, config)

	// group results by agent type
	return groupResults(responses), nil
}

// collectTestRequests gathers all test requests based on configuration
func collectTestRequests(registry *testdata.TestRegistry, prv provider.Provider, config *testConfig) []testRequest {

View on GitHub (pinned to ea665308ba)

Solutions

1. Inspect the wrapped %w cause — it names the underlying load/parse failure; fix that file first.
2. Validate all testdata registry files against the expected TestRegistry schema (a recently added/edited case is the usual culprit).
3. Rebuild the binary so go:embed picks up the current testdata contents.
4. Pass a known-good custom registry via the WithCustomRegistry option to isolate whether the builtin data or the loader is at fault.
5. git diff the testdata directory to find the most recent registry change.

Example fix

// before: builtin registry contains a malformed case added in testdata
registry, err := testdata.LoadBuiltinRegistry() // err: cannot unmarshal ...
// after: fix the YAML/JSON in testdata, or fall back to a custom registry
if err != nil {
    registry, err = loadCustomRegistry("testdata/custom-registry.yaml")
    if err != nil {
        return ProviderTestResults{}, fmt.Errorf("failed to load test registry: %w", err)
    }
}

registry, err := testdata.LoadBuiltinRegistry()
if err != nil {
    return fmt.Errorf("builtin registry unusable, fix testdata before running tester: %w", err)
}
if len(registry.Suites) == 0 {
    return fmt.Errorf("builtin registry loaded but is empty — check go:embed of testdata")
}

func registryUsable(r *testdata.TestRegistry) bool {
    return r != nil
}

results, err := tester.TestProvider(ctx, prv)
if err != nil {
    var loadErr error
    if errors.As(err, &loadErr) && strings.Contains(err.Error(), "failed to load test registry") {
        log.Fatalf("registry load failed, underlying cause: %v", errors.Unwrap(err))
    }
    return err
}

When it happens

Trigger: Calling tester.TestProvider(ctx, prv, ...) without WithCustomRegistry when LoadBuiltinRegistry() fails — typically because the embedded registry YAML/JSON is missing, corrupt, or fails schema validation (registry, err = testdata.LoadBuiltinRegistry(); runner.go:44-47).

Common situations: A developer added a new test case to the builtin registry with invalid schema; build tags or embed directives excluding testdata files so the embedded FS is empty; a malformed group/test definition committed to testdata; running a stale binary built before testdata was populated.

Related errors

• no tests to execute
• token validation disabled with default salt
• failed to switch provider: %w
• knowledge: embedding provider is not configured
• test case has no prompt or messages

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/ed19258c0543629f. Report an issue: GitHub.