vxcontrol/pentagi · error

test case has no prompt or messages

Error message

test case has no prompt or messages

What it means

executeTest dispatches a test case in priority order: messages-based cases, streaming callback cases, then simple prompt cases. If a test case defines none of these — empty prompt and no messages — the runner cannot construct any LLM call and returns this error instead of executing the request.

Source

Thrown at backend/pkg/providers/tester/runner.go:301

					if err != nil {
						break
					}
				}
			}
		}
	case len(req.testCase.Messages()) > 0:
		// messages without tools
		response, err = req.provider.CallEx(
			ctx,
			req.agentType,
			req.testCase.Messages(),
			req.testCase.StreamingCallback(),
		)
	case req.testCase.Prompt() != "":
		// simple prompt
		response, err = req.provider.Call(ctx, req.agentType, req.testCase.Prompt())
	default:
		return testdata.TestResult{}, fmt.Errorf("test case has no prompt or messages")
	}

	latency := time.Since(startTime)

	if err != nil {
		return testdata.TestResult{
			ID:          req.testCase.ID(),
			Name:        req.testCase.Name(),
			Type:        req.testCase.Type(),
			Group:       req.testCase.Group(),
			Capability:  req.testCase.Capability(),
			Success:     false,
			Unsupported: isUnsupportedCapabilityError(err),
			Error:       err,
			Latency:     latency,
		}, nil
	}

View on GitHub (pinned to ea665308ba)

Solutions

1. Implement Prompt() (or Messages()) on the custom TestCase to return the actual test prompt.
2. Fix the registry entry so the prompt/messages field is populated and deserialized with the correct key.
3. Add a registry-load-time validation that rejects test cases with neither prompt nor messages so the error surfaces at load, not at execution.
4. Check for struct-tag or YAML/JSON key mismatches that make Prompt() return "" after unmarshaling.

Example fix

// before
type myCase struct{ id string }
func (c *myCase) Prompt() string { return "" } // forgot to set prompt
// after
type myCase struct{ id, prompt string }
func (c *myCase) Prompt() string { return c.prompt } // prompt: "list the tools available to you"

// validate test cases at registry load time
for _, tc := range cases {
    if tc.Prompt() == "" && len(tc.Messages()) == 0 {
        return fmt.Errorf("test case %q has no prompt or messages", tc.ID())
    }
}

func testCaseRunnable(tc testdata.TestCase) bool {
    return tc.Prompt() != "" || len(tc.Messages()) > 0
}

res, err := executeTest(ctx, req)
if err != nil && strings.Contains(err.Error(), "has no prompt or messages") {
    return testdata.TestResult{ID: req.testCase.ID(), Name: req.testCase.Name(),
        Error: fmt.Errorf("malformed test case %q: %w", req.testCase.ID(), err)}
}

When it happens

Trigger: A TestCase registered in the registry (builtin or custom) whose Messages() is empty AND Prompt() returns "" — executeTest falls through its switch (runner.go:295-303) and hits the default branch. Usually a custom test case implemented with neither field set, or a registry entry with a missing/empty prompt key.

Common situations: A hand-written TestCase added to a custom registry where the author set only the name/ID metadata and forgot the prompt; a YAML/JSON registry entry with an empty prompt field; a refactor of a test case that moved the prompt into a field not read by Prompt(); deserialization silently dropping the prompt key due to a tag mismatch.

Related errors

• invalid provider config: %w
• unsupported agent type: %s
• failed to load test registry: %w
• no tests to execute
• invalid UserStatus: %s

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/9697af4d3985dc86. Report an issue: GitHub.