sgl-project/sglang · error · ValueError

labels() cannot be called on an already-labeled metric.

Error message

labels() cannot be called on an already-labeled metric.

What it means

RayPrometheusMetric.labels() is designed to be called once on a fresh (unlabeled) metric; it returns a labeled clone and marks it. Calling .labels() again on that clone would stack/duplicate labels and is rejected.

Source

Thrown at python/sglang/srt/observability/ray_wrappers.py:118

        return tuple(labels)

    def _build_tags(self, *labels: str, **labelskwargs: str) -> dict:
        if labels:
            # The trailing entry of ``_tag_keys`` is always ``ReplicaId`` which we
            # populate ourselves; positional args fill the preceding keys only.
            expected = len(self.metric._tag_keys) - 1
            if len(labels) != expected:
                raise ValueError(
                    "Number of labels must match the number of tag keys. "
                    f"Expected {expected}, got {len(labels)}"
                )
            labelskwargs.update(zip(self.metric._tag_keys, labels))
        labelskwargs["ReplicaId"] = _get_replica_id() or ""
        return {k: v if isinstance(v, str) else str(v) for k, v in labelskwargs.items()}

    def labels(self, *labels: str, **labelskwargs: str) -> RayPrometheusMetric:
        if self._is_labeled:
            raise ValueError("labels() cannot be called on an already-labeled metric.")
        clone = copy.copy(self)
        clone._tags = self._build_tags(*labels, **labelskwargs)
        clone._is_labeled = True
        return clone

    @staticmethod
    def _coerce_positive_boundaries(buckets):
        # Ray (gRPC OpenCensus / OpenTelemetry export) rejects boundaries
        # <= 0. sglang ships several histograms whose lowest bucket is 0.0
        # (e.g. queue_time, e2e latency). Silently drop those so we never
        # break engine startup when the metrics backend is Ray.
        if not buckets:
            return []
        return [b for b in buckets if b > 0]

    @staticmethod
    def _get_sanitized_opentelemetry_name(name: str) -> str:
        """Replace characters Ray's OTel-backed metric name validator rejects.

View on GitHub (pinned to 0132848349)

Solutions

1. Call labels() once on the original unlabeled metric and reuse the returned child for .observe/.inc etc.
2. Keep a reference to the unlabeled parent if you need differently-labeled children
3. Restructure helper functions so they return the labeled child instead of re-labeling

Example fix

# before
child = parent.labels('m1')
child = child.labels('m1')  # ValueError
# after
child = parent.labels('m1')
child.inc()

assert not metric._is_labeled, 'already labeled; reuse the labeled child'
labeled = metric.labels('model', 'gpu')

When it happens

Trigger: Chaining metric.labels(...).labels(...) or reusing a metric object that a factory already labeled.

Common situations: Shared metric singletons that get labeled at startup then again per-request; refactors that moved a labels() call into a helper invoked twice.

Related errors

• RayPrometheusMetric requires Ray to be installed. Install wi
• Number of labels must match the number of tag keys. Expected
• Missing previous frame for delta payload
• kernel dispatch requires at least one tensor argument
• Ray is required for --use-ray mode. Install it with: pip ins

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e72a9db32aca3650. Report an issue: GitHub.