sgl-project/sglang · error · ValueError

No trace files found for profile_id

Error message

No trace files found for profile_id: {self.profile_id}

What it means

Profile merger error: no torch-profiler trace files matching the given profile_id were found in the expected output directory. Verify the profiling run completed and that the profile_id/output directory are correct before merging.

Solutions

  1. Verify the trace directory exists and contains *.json/.gz trace files for the profile_id
  2. Ensure output_dir and profile_id passed to the merger match those used when starting profiling
  3. Confirm the profiling session actually started (check logs for 'Profiling starts') and workers flushed traces before merging

Example fix

# before
merger = ProfileTraceMerger(output_dir='/tmp/wrong', profile_id='p1').merge_chrome_traces()
# after
merger = ProfileTraceMerger(output_dir=real_output_dir, profile_id='p1').merge_chrome_traces()
Defensive patterns

Strategy: validation

Validate before calling

from pathlib import Path
traces = list(Path(output_dir).rglob(f'*{profile_id}*trace*'))
assert traces, f'no traces for {profile_id}, not starting merge'

Type guard

null

Try / catch

try:
    merger.merge_chrome_traces()
except ValueError as e:
    if 'No trace files' in str(e):
        logger.error('profiling produced no output; rerun with wider window')

Prevention

When it happens

Trigger: Calling merge_chrome_traces with a profile_id whose directory is empty/missing, traces written to a different output_dir, or profiling never actually produced a trace (failed start, wrong worker).

Common situations: Merging after a profiler run where workers crashed, output_dir misconfigured between start and merge, or profile_id typo'd/mismatched with the folder name.

Related errors


AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/765a490006d9488c. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/utils/profile_merger.py:50

            "pp_rank": 10_000,
            "tp_rank": 100,
        }

        # PID threshold for sort_index updates (only update for system PIDs < 1000)
        self.pid_sort_index_threshold = 1000

    def merge_chrome_traces(self) -> str:
        """Merge Chrome traces from all ranks into a single trace.

        Returns:
            Path to merged trace file.

        Raises:
            ValueError: If no trace files found.
        """
        trace_files = self._discover_trace_files()
        if not trace_files:
            raise ValueError(f"No trace files found for profile_id: {self.profile_id}")

        logger.info(f"Found {len(trace_files)} trace files to merge")

        merged_trace = {"traceEvents": []}
        all_device_properties = []

        for trace_file in sorted(trace_files, key=self._get_rank_sort_key):
            rank_info = self._extract_rank_info(trace_file)
            logger.info(f"Processing {trace_file} with rank info: {rank_info}")

            output = self._handle_file(trace_file, rank_info)

            merged_trace["traceEvents"].extend(output["traceEvents"])

            if "deviceProperties" in output:
                all_device_properties.extend(output["deviceProperties"])
                del output["deviceProperties"]

View on GitHub (pinned to 0132848349)