vxcontrol/pentagi · error

test case has no prompt or messages

Error message

test case has no prompt or messages

What it means

executeTest dispatches a test case in priority order: messages-based cases, streaming callback cases, then simple prompt cases. If a test case defines none of these — empty prompt and no messages — the runner cannot construct any LLM call and returns this error instead of executing the request.

Solutions

1. Implement Prompt() (or Messages()) on the custom TestCase to return the actual test prompt.
2. Fix the registry entry so the prompt/messages field is populated and deserialized with the correct key.
3. Add a registry-load-time validation that rejects test cases with neither prompt nor messages so the error surfaces at load, not at execution.
4. Check for struct-tag or YAML/JSON key mismatches that make Prompt() return "" after unmarshaling.

Example fix

// before
type myCase struct{ id string }
func (c *myCase) Prompt() string { return "" } // forgot to set prompt
// after
type myCase struct{ id, prompt string }
func (c *myCase) Prompt() string { return c.prompt } // prompt: "list the tools available to you"

// validate test cases at registry load time
for _, tc := range cases {
    if tc.Prompt() == "" && len(tc.Messages()) == 0 {
        return fmt.Errorf("test case %q has no prompt or messages", tc.ID())
    }
}

func testCaseRunnable(tc testdata.TestCase) bool {
    return tc.Prompt() != "" || len(tc.Messages()) > 0
}

res, err := executeTest(ctx, req)
if err != nil && strings.Contains(err.Error(), "has no prompt or messages") {
    return testdata.TestResult{ID: req.testCase.ID(), Name: req.testCase.Name(),
        Error: fmt.Errorf("malformed test case %q: %w", req.testCase.ID(), err)}
}

When it happens

Trigger: A TestCase registered in the registry (builtin or custom) whose Messages() is empty AND Prompt() returns "" — executeTest falls through its switch (runner.go:295-303) and hits the default branch. Usually a custom test case implemented with neither field set, or a registry entry with a missing/empty prompt key.

Common situations: A hand-written TestCase added to a custom registry where the author set only the name/ID metadata and forgot the prompt; a YAML/JSON registry entry with an empty prompt field; a refactor of a test case that moved the prompt into a field not read by Prompt(); deserialization silently dropping the prompt key due to a tag mismatch.

Related errors

• builtin function not found
• definitions and handlers must have the same length
• failed to load test registry
• handler for function
• invalid provider config

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/9697af4d3985dc86. Report an issue: GitHub.

Appendix: source

Thrown at backend/pkg/providers/tester/runner.go:301

					if err != nil {
						break
					}
				}
			}
		}
	case len(req.testCase.Messages()) > 0:
		// messages without tools
		response, err = req.provider.CallEx(
			ctx,
			req.agentType,
			req.testCase.Messages(),
			req.testCase.StreamingCallback(),
		)
	case req.testCase.Prompt() != "":
		// simple prompt
		response, err = req.provider.Call(ctx, req.agentType, req.testCase.Prompt())
	default:
		return testdata.TestResult{}, fmt.Errorf("test case has no prompt or messages")
	}

	latency := time.Since(startTime)

	if err != nil {
		return testdata.TestResult{
			ID:          req.testCase.ID(),
			Name:        req.testCase.Name(),
			Type:        req.testCase.Type(),
			Group:       req.testCase.Group(),
			Capability:  req.testCase.Capability(),
			Success:     false,
			Unsupported: isUnsupportedCapabilityError(err),
			Error:       err,
			Latency:     latency,
		}, nil
	}

View on GitHub (pinned to ea665308ba)