microsoft/qlib · error · ValueError

source_dir and target_dir cannot be None

Error message

source_dir and target_dir cannot be None

What it means

Raised by BaseNormalize constructor in scripts/data_collector/base.py when source_dir or target_dir is falsy (None or empty string). The normalize pipeline needs both an input directory of collected files and an output directory and refuses to continue without them.

Source

Thrown at scripts/data_collector/base.py:275

        """

        Parameters
        ----------
        source_dir: str or Path
            The directory where the raw data collected from the Internet is saved
        target_dir: str or Path
            Directory for normalize data
        normalize_class: Type[YahooNormalize]
            normalize class
        max_workers: int
            Concurrent number, default is 16
        date_field_name: str
            date field name, default is date
        symbol_field_name: str
            symbol field name, default is symbol
        """
        if not (source_dir and target_dir):
            raise ValueError("source_dir and target_dir cannot be None")
        self._source_dir = Path(source_dir).expanduser()
        self._target_dir = Path(target_dir).expanduser()
        self._target_dir.mkdir(parents=True, exist_ok=True)
        self._date_field_name = date_field_name
        self._symbol_field_name = symbol_field_name
        self._end_date = kwargs.get("end_date", None)
        self._max_workers = max_workers
        self.interval = kwargs.get("interval", "1d")

        self._normalize_obj = normalize_class(
            date_field_name=date_field_name, symbol_field_name=symbol_field_name, **kwargs
        )

    def format_data(self, df: pd.DataFrame):
        if self.interval == "1d":
            try:
                pd.to_datetime(df.iloc[-1]["date"], format="%Y-%m-%d", errors="raise")
            except Exception:

View on GitHub (pinned to 79633dd950)

Solutions

1. Supply both flags: --source_dir <dir of downloaded raw files> --target_dir <dir for normalized files>.
2. Echo/inspect the shell variables before invoking the CLI to ensure neither is empty.
3. Check the collector's README example command; some collectors need extra args (e.g. --qlib_data_1d_dir) but source/target are always mandatory.

Example fix

# before
python collector.py normalize_data --source_dir $SRC --normalize_dir /data/nor

# after
python collector.py normalize_data --source_dir /data/source/1d --normalize_dir /data/nor

if not source_dir or not target_dir:
    raise SystemExit("--source_dir and --target_dir (or --normalize_dir) are required")

When it happens

Trigger: Invoking a data_collector tool's normalize_data command without --source_dir or --target_dir, or passing an empty string for either (the check `if not (source_dir and target_dir)` treats '' the same as None).

Common situations: Copy-pasting the collector command from docs of a different collector whose argument names differ; shell variable that expands to empty (e.g. $SOURCE_DIR unset); running download_data phase only and assuming normalize infers default directories.

Related errors

• interval error: {self.interval}
• interval error: {self.interval}
• This type of input is not supported
• response status: {_status}, url={url}
• Unsupported freq: {freq}

AI-assisted analysis of microsoft/qlib@79633dd950 (2026-08-15). Data as JSON: /api/errors/2adc03c389593f8c. Report an issue: GitHub.