{"record":{"id":"2e30ba9d3433e617","repo":"openzipkin/zipkin","slug":"credential-refresh-thread-didn-t-start","errorCode":null,"errorMessage":"credential refresh thread didn't start","messagePattern":"credential refresh thread didn't start","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"zipkin-server/src/main/java/zipkin2/server/internal/elasticsearch/ZipkinElasticsearchStorageConfiguration.java","lineNumber":165,"sourceCode":"    if (isEmpty(es.getUsername()) || isEmpty(es.getPassword())) {\n      return new BasicCredentials();\n    }\n    return new BasicCredentials(es.getUsername(), es.getPassword());\n  }\n\n  @Bean(destroyMethod = \"shutdown\") @Qualifier(QUALIFIER) @Conditional(DynamicRefreshRequired.class)\n  ScheduledExecutorService dynamicCredentialsScheduledExecutorService(\n    @Value(\"${\" + CREDENTIALS_FILE + \"}\") String credentialsFile,\n    @Value(\"${\" + CREDENTIALS_REFRESH_INTERVAL + \"}\") Integer credentialsRefreshInterval,\n    @Qualifier(QUALIFIER) BasicCredentials basicCredentials) throws IOException {\n    ScheduledExecutorService ses = Executors.newSingleThreadScheduledExecutor(\n      new NamedThreadFactory(\"zipkin-load-es-credentials\"));\n    DynamicCredentialsFileLoader credentialsFileLoader =\n      new DynamicCredentialsFileLoader(basicCredentials, credentialsFile);\n    credentialsFileLoader.updateCredentialsFromProperties();\n    ScheduledFuture<?> future = ses.scheduleAtFixedRate(credentialsFileLoader,\n        0, credentialsRefreshInterval, TimeUnit.SECONDS);\n    if (future.isDone()) throw new RuntimeException(\"credential refresh thread didn't start\");\n    return ses;\n  }\n\n  @Bean @Qualifier(QUALIFIER) @ConditionalOnSelfTracing\n  Consumer<ClientOptionsBuilder> esTracing(Optional<HttpTracing> maybeHttpTracing) {\n    if (maybeHttpTracing.isEmpty()) {\n      // TODO: is there a special cased empty consumer we can use here? I suspect debug is cluttered\n      // Alternatively, check why we would ever get here if ConditionalOnSelfTracing matches\n      return client -> {\n      };\n    }\n\n    HttpTracing httpTracing = maybeHttpTracing.get().clientOf(\"elasticsearch\");\n    SpanCustomizer spanCustomizer = CurrentSpanCustomizer.create(httpTracing.tracing());\n\n    return client -> {\n      client.decorator((delegate, ctx, req) -> {\n        // We only need the name if it's available and can unsafely access the partially filled log.","sourceCodeStart":147,"sourceCodeEnd":183,"githubUrl":"https://github.com/openzipkin/zipkin/blob/878ce2a1fad54ca941d17fdcf2e1d924b148eb1f/zipkin-server/src/main/java/zipkin2/server/internal/elasticsearch/ZipkinElasticsearchStorageConfiguration.java#L147-L183","documentation":"ZipkinElasticsearchStorageConfiguration.dynamicCredentialsScheduledExecutorService throws RuntimeException ('credential refresh thread didn't start') when the ScheduledFuture returned by scheduleAtFixedRate for the credentials-file loader is already done. For a fixed-rate task with non-zero period, immediate completion means the first execution threw (e.g. file unreadable or invalid), which would silently disable all future refreshes, so the bean fails fast at startup.","triggerScenarios":"zipkin.storage.elasticsearch.credentials-file is configured and the very first run of DynamicCredentialsFileLoader throws (missing file, unreadable permissions, missing/empty username or password keys), marking the scheduled task permanently done; the isDone() check then fires.","commonSituations":"Credentials file not yet mounted when the server boots (race with k8s secrets/vault), file permissions deny the JVM read access, or file content invalid per ensureNotEmptyOrNull; the underlying first-run exception is the root cause.","solutions":["Inspect the earlier log line/stack trace for the first-run failure of DynamicCredentialsFileLoader and fix that root cause","Ensure the credentials file exists, is readable by the Zipkin process, and contains non-empty username and password keys","Delay server start until secret mounts are present (e.g. k8s secret volumes, initContainer) or fix the path in zipkin.storage.elasticsearch.credentials-file"],"exampleFix":"# before\nzipkin.storage.elasticsearch.credentials-file=/etc/secrets/escreds # file missing at boot\n\n# after\n# mount the secret before start; verify with: cat /etc/secrets/escreds\n# containing:\n#   username=es-user\n#   password=secret\nzipkin.storage.elasticsearch.credentials-file=/etc/secrets/escreds","handlingStrategy":"try-catch","validationCode":"Path f = Path.of(credentialsFile);\nif (!Files.isReadable(f)) throw new IllegalStateException(\"credentials file not readable: \" + f);","typeGuard":null,"tryCatchPattern":"wrap bean creation in try-catch, log the first-execution cause from the ScheduledFuture (future.get() throws ExecutionException with the root cause), fix the file, and restart","preventionTips":["Order deployment steps so secret mounts/vault renders complete before the Zipkin process starts","Health-check credentials file presence and readability in a startup probe"],"tags":["zipkin","elasticsearch","credentials","scheduler","startup"],"backgroundTag":null,"analyzedSha":"878ce2a1fad54ca941d17fdcf2e1d924b148eb1f","analyzedAt":"2026-08-14T15:17:09.895Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}