What causes "DataQueue can not activate twice." in microsoft/qlib?

Drop the manual `activate()` call and rely on the with-block: `with DataQueue(...) as q: for item in q: ...`.

microsoft/qlib · error · ValueError

DataQueue can not activate twice.

Error message

DataQueue can not activate twice.

What it means

ValueError from `DataQueue.activate` (qlib/rl/utils/data_queue.py:140). A DataQueue launches exactly one daemon producer thread on first activation; `activate` is guarded by the `_activated` flag and refuses a second call. In normal usage the `with` block (`__enter__`) activates it, so manual activation plus a with-block causes the double fire.

Source

Thrown at qlib/rl/utils/data_queue.py:140

            try:
                return self._queue.get(block=block, timeout=timeout)
            except Empty:
                if self._done.value:
                    raise StopIteration  # pylint: disable=raise-missing-from

    def put(self, obj: Any, block: bool = True, timeout: int | None = None) -> None:
        self._queue.put(obj, block=block, timeout=timeout)

    def mark_as_done(self) -> None:
        with self._done.get_lock():
            self._done.value = 1

    def done(self) -> int:
        return self._done.value

    def activate(self) -> DataQueue:
        if self._activated:
            raise ValueError("DataQueue can not activate twice.")
        thread = threading.Thread(target=self._producer, daemon=True)
        thread.start()
        self._activated = True
        return self

    def __del__(self) -> None:
        _logger.debug(f"__del__ of {__name__}.DataQueue")
        self.cleanup()

    def __iter__(self) -> Generator[Any, None, None]:
        if not self._activated:
            raise ValueError(
                "Need to call activate() to launch a daemon worker "
                "to produce data into data queue before using it. "
                "You probably have forgotten to use the DataQueue in a with block.",
            )
        return self._consumer()

View on GitHub (pinned to 79633dd950)

Solutions

Drop the manual `activate()` call and rely on the with-block: `with DataQueue(...) as q: for item in q: ...`.
If managing lifecycle manually, check `q._activated` semantics via the public contract: activate once, and create a new DataQueue instance for each run.
Restructure code so there is exactly one owner of activation (either the context manager or your code, never both).

Example fix

// before
q = DataQueue(gen).activate()
with q:  # __enter__ activates again -> ValueError
    for x in q: ...
// after
with DataQueue(gen) as q:  # single automatic activation
    for x in q: ...

Defensive patterns

Strategy: validation

Validate before calling

def start_queue(q):
    if getattr(q, "_activated", False):
        raise RuntimeError("DataQueue already activated; create a new instance per run")
    return q.activate()

Type guard

def queue_is_activated(q) -> bool:
    return bool(getattr(q, "_activated", False))

Try / catch

try:
    with DataQueue(gen) as q:
        pass
except ValueError as e:
    if "activate twice" in str(e):
        q = DataQueue(gen)  # fresh instance, single activation
    else:
        raise

Prevention

Never call activate() manually when using the with-block; the context manager owns activation.
Create a new DataQueue for each training run or epoch.
Keep exactly one lifecycle owner for the queue in your codebase.

When it happens

Trigger: Calling `queue.activate()` explicitly and then entering `with queue:` (which calls activate again); or a custom lifecycle manager that activates the queue on setup and again on start.

Common situations: Integrating DataQueue into a custom training loop where activation was already handled by a wrapper; copy-pasted setup code that activates 'for safety'; reusing one DataQueue object across two training runs without recreating it.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/a3a4a238a4b53b65. Report an issue: GitHub.