gocolly/colly · error

cannot call duplicate Queue.Run

Error message

cannot call duplicate Queue.Run

What it means

Queue.Run blocks while draining the queue, so running it twice concurrently on the same Queue would double-consume requests. The Queue guards this with a mutex and panics with 'cannot call duplicate Queue.Run' if q.running is already true. It is a programmer-error guard, not a recoverable library error.

Source

Thrown at queue/queue.go:137

	if err != nil {
		return err
	}
	return q.storage.AddRequest(d)
}

// Size returns the size of the queue
func (q *Queue) Size() (int, error) {
	return q.storage.QueueSize()
}

// Run starts consumer threads and calls the Collector
// to perform requests. Run blocks while the queue has active requests
// The given Storage must not be used directly while Run blocks.
func (q *Queue) Run(c *colly.Collector) error {
	q.mut.Lock()
	if q.wake != nil && q.running == true {
		q.mut.Unlock()
		panic("cannot call duplicate Queue.Run")
	}
	q.wake = make(chan struct{})
	q.running = true
	q.mut.Unlock()

	requestc := make(chan *colly.Request)
	complete, errc := make(chan struct{}), make(chan error, 1)
	for i := 0; i < q.Threads; i++ {
		go independentRunner(requestc, complete)
	}
	go q.loop(c, requestc, complete, errc)
	defer close(requestc)
	return <-errc
}

// Stop will stop the running queue
func (q *Queue) Stop() {
	q.mut.Lock()

View on GitHub (pinned to 17d1d6ca92)

Solutions

1. Ensure only one goroutine calls Run per Queue; wait for Run to return before calling it again
2. If you need concurrent draining, create separate Queue instances per worker group
3. Recover from the panic only at a top-level boundary, then restructure startup so Run is called exactly once
4. Queue new URLs before Run starts, or use Queue.AddURL/append before a single Run

Example fix

// before
go queue.Run(c)
...
go queue.Run(c) // panics: duplicate Run
// after
if err := queue.Run(c); err != nil { // single blocking call
    log.Fatal(err)
}

// guard before calling Run: track invocation yourself
var runStarted int32
func safeRun(q *queue.Queue, c *colly.Collector) error {
    if !atomic.CompareAndSwapInt32(&runStarted, 0, 1) {
        return errors.New("Run already invoked")
    }
    return q.Run(c)
}

func drain(q *queue.Queue, c *colly.Collector) (err error) {
    defer func() {
        if r := recover(); r != nil {
            if s, ok := r.(string); ok && strings.Contains(s, "duplicate Queue.Run") {
                err = errors.New("queue already running")
                return
            }
            panic(r)
        }
    }()
    return q.Run(c)
}

When it happens

Trigger: Calling q.Run(collector) a second time (e.g. from another goroutine or after an initial Run plus re-invocation) while the first Run is still active — q.wake != nil && q.running == true.

Common situations: Restarting a queue after adding new URLs without waiting for the first Run to return; accidentally starting Run in multiple goroutines; framework callbacks re-invoking Run; mistaking Run for non-blocking and calling it in a loop.

Related errors

• ErrQueueFull

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/b273fee72f8f679f. Report an issue: GitHub.