What causes "cannot clear memory as `num_states==0`" in microsoft/qlib?

Set num_states > 0 at dataset construction if state memory is required.

microsoft/qlib · error · ValueError

cannot clear memory as `num_states==0`

Error message

cannot clear memory as `num_states==0`

What it means

clear_memory zeroes the RL dataset's state-memory tensor between epochs/episodes. Like assign_data, it is only meaningful when num_states > 0; with num_states == 0 there is no memory to clear and the call raises ValueError, signaling the dataset was configured for memoryless (supervised) use.

Source

Thrown at qlib/contrib/data/dataset.py:251

        obj._daily_index = self._daily_index[mask]
        return obj

    def restore_index(self, index):
        return self._index[index]

    def restore_daily_index(self, daily_index):
        return pd.Index(self._daily_index.loc[daily_index])

    def assign_data(self, index, vals):
        if self.num_states == 0:
            raise ValueError("cannot assign data as `num_states==0`")
        if isinstance(vals, torch.Tensor):
            vals = vals.detach().cpu().numpy()
        self._memory[index] = vals

    def clear_memory(self):
        if self.num_states == 0:
            raise ValueError("cannot clear memory as `num_states==0`")
        self._memory[:] = 0

    def train(self):
        """enable traning mode"""
        self.batch_size, self.n_samples, self.drop_last, self.shuffle = self.params

    def eval(self):
        """enable evaluation mode"""
        self.batch_size = -1
        self.n_samples = None
        self.drop_last = False
        self.shuffle = False

    def _get_slices(self):
        if self.batch_size < 0:  # daily sampling
            slices = self._daily_slices.copy()
            batch_size = -1 * self.batch_size
        else:  # normal sampling

View on GitHub (pinned to 79633dd950)

Solutions

Set num_states > 0 at dataset construction if state memory is required.
Condition the call: if ds.num_states > 0: ds.clear_memory().
Audit config plumbing to confirm num_states reaches the dataset constructor with the intended value.

Example fix

# before
for ep in episodes:
    train_episode()
    ds.clear_memory()  # -> ValueError when num_states==0
# after
for ep in episodes:
    train_episode()
    if ds.num_states > 0:
        ds.clear_memory()

Defensive patterns

Strategy: validation

Validate before calling

if ds.num_states > 0:
    ds.clear_memory()
# else: no memory allocated, nothing to clear

Type guard

def has_state_memory(ds) -> bool:
    return getattr(ds, 'num_states', 0) > 0

Try / catch

try:
    ds.clear_memory()
except ValueError as e:
    if 'num_states==0' in str(e):
        pass  # expected for memoryless datasets
    else:
        raise

Prevention

Make memory lifecycle calls (assign_data/clear_memory) conditional on num_states > 0.
Log num_states once at dataset construction to catch config plumbing mistakes early.

When it happens

Trigger: An RL training loop unconditionally calling dataset.clear_memory() at episode end while the dataset was constructed with num_states=0.

Common situations: Shared training scripts used for both supervised and RL runs; num_states left at its default because the config key was renamed or nested incorrectly.

Related errors

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/a7188e50f0c0a4f5. Report an issue: GitHub.