microsoft/qlib · error · ValueError

Need to call activate() to launch a daemon worker to produce

Error message

Need to call activate() to launch a daemon worker to produce data into data queue before using it. You probably have forgotten to use the DataQueue in a with block.

What it means

ValueError from `DataQueue.__iter__` (qlib/rl/utils/data_queue.py:152). Iterating a DataQueue before its daemon producer thread was started is forbidden; activation happens in `__enter__`, so this almost always means the queue was used outside its `with` block.

Source

Thrown at qlib/rl/utils/data_queue.py:152

    def done(self) -> int:
        return self._done.value

    def activate(self) -> DataQueue:
        if self._activated:
            raise ValueError("DataQueue can not activate twice.")
        thread = threading.Thread(target=self._producer, daemon=True)
        thread.start()
        self._activated = True
        return self

    def __del__(self) -> None:
        _logger.debug(f"__del__ of {__name__}.DataQueue")
        self.cleanup()

    def __iter__(self) -> Generator[Any, None, None]:
        if not self._activated:
            raise ValueError(
                "Need to call activate() to launch a daemon worker "
                "to produce data into data queue before using it. "
                "You probably have forgotten to use the DataQueue in a with block.",
            )
        return self._consumer()

    def _consumer(self) -> Generator[Any, None, None]:
        while True:
            try:
                yield self.get()
            except StopIteration:
                _logger.debug("Data consumer timed-out from get.")
                return

    def _producer(self) -> None:
        # pytorch dataloader is used here only because we need its sampler and multi-processing
        from torch.utils.data import DataLoader, Dataset  # pylint: disable=import-outside-toplevel

View on GitHub (pinned to 79633dd950)

Solutions

Wrap usage in the context manager: `with DataQueue(producer) as q: consume(q)`.
If you must manage it manually, call `q = q.activate()` once before the first `for` loop (and never combine with the with-block, see the double-activate error).
Keep creation and consumption in the same scope so the with-block visibly encloses every iteration site.

Example fix

// before
q = DataQueue(producer)
for item in q:  # not activated -> ValueError
    ...
// after
with DataQueue(producer) as q:
    for item in q:
        ...

Defensive patterns

Strategy: validation

Validate before calling

def iter_data_queue(q):
    if not getattr(q, "_activated", False):
        raise RuntimeError("activate the DataQueue (or use a with-block) before iterating")
    return iter(q)

Type guard

def queue_ready(q) -> bool:
    return bool(getattr(q, "_activated", False))

Try / catch

try:
    for item in q:
        process(item)
except ValueError as e:
    if "forgotten to use the DataQueue in a with block" in str(e):
        with q:
            for item in q:
                process(item)
    else:
        raise

Prevention

Always consume a DataQueue inside `with DataQueue(...) as q:`.
Pass the queue to helpers already inside the with-block, not before.
In notebooks, keep creation and iteration in the same cell or a single with-block.

When it happens

Trigger: `for x in DataQueue(generator):` without a surrounding `with`, or iterating a queue created earlier whose with-block has already exited; passing an unactivated DataQueue to code that iterates it.

Common situations: Refactoring training code and losing the with-statement; helper functions receiving the queue and iterating it while the caller forgot to enter the context; notebooks where cells run out of order relative to the with-block.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/ec1fa12e20b9c973. Report an issue: GitHub.