vxcontrol/pentagi · error

task stop timeout

Error message

task stop timeout

What it means

Stop waits for all running tasks (fw.taskWG) to finish behind a pre-configured timeout timer; if taskWG.Wait() hasn't completed when the timer fires, it returns 'task stop timeout' and leaks nothing but leaves the tasks still running/unterminated. It signals that graceful stopping of the flow's tasks exceeded the allowed grace period.

Source

Thrown at backend/pkg/controller/flow.go:844

	ctx, span := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "controller.flowWorker.Stop")
	defer span.End()

	fw.taskMX.Lock()
	defer fw.taskMX.Unlock()

	fw.taskST()
	done := make(chan struct{})
	timer := time.NewTimer(stopTaskTimeout)
	defer timer.Stop()

	go func() {
		fw.taskWG.Wait()
		close(done)
	}()

	select {
	case <-timer.C:
		return fmt.Errorf("task stop timeout")
	case <-done:
		return nil
	}
}

func (fw *flowWorker) Rename(ctx context.Context, title string) error {
	fw.flowCtx.Provider.SetTitle(title)

	flow, err := fw.flowCtx.DB.UpdateFlowTitle(ctx, database.UpdateFlowTitleParams{
		ID:    fw.flowCtx.FlowID,
		Title: title,
	})
	if err != nil {
		return fmt.Errorf("failed to rename flow %d: %w", fw.flowCtx.FlowID, err)
	}

	containers, err := fw.flowCtx.DB.GetFlowContainers(ctx, fw.flowCtx.FlowID)
	if err != nil {

View on GitHub (pinned to ea665308ba)

Solutions

1. Find which task ignored cancellation: ensure every task/goroutine honors ctx.Done() and uses ctx-scoped Docker/LLM calls.
2. Increase the stop grace timeout if tasks are legitimately long but cancellable.
3. Force-kill the flow's Docker containers to unblock stuck exec sessions, then retry Stop.
4. Add per-tool timeouts so no single tool call can hang a task indefinitely.

Example fix

// before
out, err := exec.CommandContext(context.Background(), cmd) // ignores stop
// after
out, err := exec.CommandContext(ctx, cmd) // aborts when flow stops

// ensure no task can outlive the stop grace: every tool/LLM call must take ctx
func tasksCancellable() bool { /* verify exec/LLM calls use fw.ctx-derived contexts */ return true }

func isStopTimeout(err error) bool {
    return err != nil && err.Error() == "task stop timeout"
}

if err := worker.Stop(ctx); err != nil {
    if isStopTimeout(err) {
        // escalate: kill flow containers, then mark worker stopped
        _ = killFlowContainers(ctx, worker.FlowID())
        return worker.ForceCleanup(ctx)
    }
    return err
}

When it happens

Trigger: Calling fw.Stop while an assistant task is stuck: a long-running Docker exec command, an LLM streaming call with no deadline, a blocked channel write in the task, or a deadlock in task code — taskWG.Wait() does not return before the timer expires.

Common situations: Agent waiting on a hung container command (no timeout configured); LLM provider network stall without ctx deadline; task goroutine blocked publishing to a full channel; very long tool invocation that ignores ctx cancellation.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• flow %d input processing timeout: %w
• bearer scheme must be used
• token can't be empty
• token validation disabled with default salt
• token is invalid

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/504fb23eca48ebf9. Report an issue: GitHub.