openzipkin/zipkin · critical · RuntimeException

credential refresh thread didn't start

Error message

credential refresh thread didn't start

What it means

ZipkinElasticsearchStorageConfiguration.dynamicCredentialsScheduledExecutorService throws RuntimeException ('credential refresh thread didn't start') when the ScheduledFuture returned by scheduleAtFixedRate for the credentials-file loader is already done. For a fixed-rate task with non-zero period, immediate completion means the first execution threw (e.g. file unreadable or invalid), which would silently disable all future refreshes, so the bean fails fast at startup.

Source

Thrown at zipkin-server/src/main/java/zipkin2/server/internal/elasticsearch/ZipkinElasticsearchStorageConfiguration.java:165

    if (isEmpty(es.getUsername()) || isEmpty(es.getPassword())) {
      return new BasicCredentials();
    }
    return new BasicCredentials(es.getUsername(), es.getPassword());
  }

  @Bean(destroyMethod = "shutdown") @Qualifier(QUALIFIER) @Conditional(DynamicRefreshRequired.class)
  ScheduledExecutorService dynamicCredentialsScheduledExecutorService(
    @Value("${" + CREDENTIALS_FILE + "}") String credentialsFile,
    @Value("${" + CREDENTIALS_REFRESH_INTERVAL + "}") Integer credentialsRefreshInterval,
    @Qualifier(QUALIFIER) BasicCredentials basicCredentials) throws IOException {
    ScheduledExecutorService ses = Executors.newSingleThreadScheduledExecutor(
      new NamedThreadFactory("zipkin-load-es-credentials"));
    DynamicCredentialsFileLoader credentialsFileLoader =
      new DynamicCredentialsFileLoader(basicCredentials, credentialsFile);
    credentialsFileLoader.updateCredentialsFromProperties();
    ScheduledFuture<?> future = ses.scheduleAtFixedRate(credentialsFileLoader,
        0, credentialsRefreshInterval, TimeUnit.SECONDS);
    if (future.isDone()) throw new RuntimeException("credential refresh thread didn't start");
    return ses;
  }

  @Bean @Qualifier(QUALIFIER) @ConditionalOnSelfTracing
  Consumer<ClientOptionsBuilder> esTracing(Optional<HttpTracing> maybeHttpTracing) {
    if (maybeHttpTracing.isEmpty()) {
      // TODO: is there a special cased empty consumer we can use here? I suspect debug is cluttered
      // Alternatively, check why we would ever get here if ConditionalOnSelfTracing matches
      return client -> {
      };
    }

    HttpTracing httpTracing = maybeHttpTracing.get().clientOf("elasticsearch");
    SpanCustomizer spanCustomizer = CurrentSpanCustomizer.create(httpTracing.tracing());

    return client -> {
      client.decorator((delegate, ctx, req) -> {
        // We only need the name if it's available and can unsafely access the partially filled log.

View on GitHub (pinned to 878ce2a1fa)

Solutions

1. Inspect the earlier log line/stack trace for the first-run failure of DynamicCredentialsFileLoader and fix that root cause
2. Ensure the credentials file exists, is readable by the Zipkin process, and contains non-empty username and password keys
3. Delay server start until secret mounts are present (e.g. k8s secret volumes, initContainer) or fix the path in zipkin.storage.elasticsearch.credentials-file

Example fix

# before
zipkin.storage.elasticsearch.credentials-file=/etc/secrets/escreds # file missing at boot

# after
# mount the secret before start; verify with: cat /etc/secrets/escreds
# containing:
#   username=es-user
#   password=secret
zipkin.storage.elasticsearch.credentials-file=/etc/secrets/escreds

Path f = Path.of(credentialsFile);
if (!Files.isReadable(f)) throw new IllegalStateException("credentials file not readable: " + f);

wrap bean creation in try-catch, log the first-execution cause from the ScheduledFuture (future.get() throws ExecutionException with the root cause), fix the file, and restart

When it happens

Trigger: zipkin.storage.elasticsearch.credentials-file is configured and the very first run of DynamicCredentialsFileLoader throws (missing file, unreadable permissions, missing/empty username or password keys), marking the scheduled task permanently done; the isDone() check then fires.

Common situations: Credentials file not yet mounted when the server boots (race with k8s secrets/vault), file permissions deny the JVM read access, or file content invalid per ensureNotEmptyOrNull; the underlying first-run exception is the root cause.

Related errors

• no {name} property in {fileName}
• empty {name} property in {fileName}
• No valid endpoints found in ES hosts: {hosts}
• dateSeparator must be empty or a single character
• Could not load Lens UI from {lensIndexHtml}

AI-assisted analysis of openzipkin/zipkin@878ce2a1fa (2026-08-14). Data as JSON: /api/errors/2e30ba9d3433e617. Report an issue: GitHub.