openzipkin/zipkin · error · IllegalArgumentException

remoteService={} unsupported due to missing column zipkin_sp

Error message

remoteService={} unsupported due to missing column zipkin_spans.remote_service_name

What it means

SelectSpansAndAnnotations.Factory.create throws IllegalArgumentException when a QueryRequest filters by remoteServiceName but the detected database schema lacks the zipkin_spans.remote_service_name column. The MySQL-v1 storage inspects the schema at startup and refuses queries it cannot express in SQL, rather than silently returning wrong results.

Source

Thrown at zipkin-storage/mysql-v1/src/main/java/zipkin2/storage/mysql/v1/SelectSpansAndAnnotations.java:71

      return new SelectSpansAndAnnotations(schema) {
        @Override
        Condition traceIdCondition(DSLContext context) {
          return schema.spanTraceIdCondition(finalTraceIdHigh, traceIdLow);
        }
      };
    }

    SelectSpansAndAnnotations create(Set<Pair> traceIdPairs) {
      return new SelectSpansAndAnnotations(schema) {
        @Override Condition traceIdCondition(DSLContext context) {
          return schema.spanTraceIdCondition(traceIdPairs);
        }
      };
    }

    SelectSpansAndAnnotations create(QueryRequest request) {
      if (request.remoteServiceName() != null && !schema.hasRemoteServiceName) {
        throw new IllegalArgumentException("remoteService=" + request.remoteServiceName()
          + " unsupported due to missing column zipkin_spans.remote_service_name");
      }
      return new SelectSpansAndAnnotations(schema) {
        @Override
        Condition traceIdCondition(DSLContext context) {
          return schema.spanTraceIdCondition(toTraceIdQuery(context, request));
        }
      };
    }
  }

  final Schema schema;

  SelectSpansAndAnnotations(Schema schema) {
    this.schema = schema;
  }

  abstract Condition traceIdCondition(DSLContext context);

View on GitHub (pinned to 878ce2a1fa)

Solutions

1. Upgrade the database schema: apply the ALTER TABLE zipkin_spans ADD COLUMN remote_service_name that ships with the zipkin MySQL DDL for 2.13+.
2. RecREATE the zipkin database from the current mysql.sql bundled with your zipkin version (acceptable when data is disposable).
3. As a stopgap, drop remoteServiceName from the query request so it no longer targets the missing column.

Example fix

-- before: old schema
-- (column zipkin_spans.remote_service_name does not exist)

-- after
ALTER TABLE zipkin_spans ADD COLUMN `remote_service_name` VARCHAR(255) DEFAULT NULL;
CREATE INDEX `zipkin_spans.remote_service_name` ON zipkin_spans(`remote_service_name`);

if (request.remoteServiceName() != null) {
  // ensure schema has zipkin_spans.remote_service_name before issuing this query
  // e.g. check the column exists or upgrade the schema first
}

try { storage.getTraces(request).execute(); } catch (IllegalArgumentException e) { if (e.getMessage().contains("remote_service_name")) { /* surface schema-upgrade instruction */ } throw e; }

When it happens

Trigger: Calling spanStore().getTraces(new QueryRequest.Builder(...).remoteServiceName("backend").build()) against a Zipkin MySQL database created with a pre-2.13 schema (before remote_service_name was added).

Common situations: Running new zipkin-server against an old MySQL schema; upgrading the zipkin jar without running the schema migration; environments where DDL is managed manually and the ALTER TABLE was missed.

Related errors

• endTs <= 0
• Expected start of dependency link object but was {}
• key == null
• keys == null
• datasource == null

AI-assisted analysis of openzipkin/zipkin@878ce2a1fa (2026-08-14). Data as JSON: /api/errors/0e8b98aa4b69c11a. Report an issue: GitHub.