zhisheng17/flink-learning · error · RuntimeException

PWD env doesn't contains yarn application id or container id

Error message

PWD env doesn't contains yarn application id or container id

What it means

KafkaReporter (Flink metrics) derives the YARN application id and container id by splitting the current working directory (PWD of the YARN container). If PWD has fewer than two path segments it cannot extract both ids, logs an error, and throws RuntimeException in open().

Source

Thrown at flink-learning-extends/flink-metrics/flink-metrics-kafka/src/main/java/org/apache/flink/metrics/kafka/KafkaReporter.java:79

	@Override
	public void open(MetricConfig config) {
		Map<String, String> envs = System.getenv();
		String clusterId = envs.get("CLUSTER_ID");
		if (clusterId != null) {
			//k8s cluster
			appId = clusterId;
			containerId = envs.get("HOSTNAME");
		} else {
			//yarn cluster
			String pwd = envs.get("PWD");
			String[] values = pwd.split(File.separator);
			if (values.length >= 2) {
				appId = values[values.length - 2];
				containerId = values[values.length - 1];
			} else {
				LOG.error("PWD env ({}) doesn't contains yarn application id or container id", pwd);
				throw new RuntimeException(
					"PWD env doesn't contains yarn application id or container id");
			}
		}

		Properties properties = System.getProperties();
		taskName = properties.getProperty("taskName", null);
		taskId = properties.getProperty("taskId", null);

		Properties props = new Properties();
		String clientIdPrefix = taskId != null ? taskId : appId;
		props.put("client.id", "flink_" + clientIdPrefix + "_metrics");
		props.put("bootstrap.servers", getString(config, BOOTSTRAP_SERVERS));
		props.put("acks", getString(config, ACKS));
		props.put("retries", getInteger(config, RETRIES));
		props.put("batch.size", getInteger(config, BATCH_SIZE));
		props.put("linger.ms", getInteger(config, LINGER_MS));
		props.put("buffer.memory", getInteger(config, BUFFER_MEMORY));
		props.put("max.request.size", getInteger(config, MAX_REQUEST_SIZE));

View on GitHub (pinned to d731cee761)

Solutions

1. Run the job on YARN so the container PWD contains the application/container id path segments.
2. Disable or swap the KafkaReporter for local/non-YARN runs (use a different reporter in local profile).
3. Patch the reporter to take appId/containerId from env vars (FLINK_APPLICATION_ID, HOSTNAME) as a fallback instead of PWD.
4. Guard the reporter initialization with a check for YARN before throwing.

Example fix

// before
String[] values = pwd.split(File.separator);
if (values.length >= 2) { ... } else { throw new RuntimeException(...); }
// after
String[] values = pwd.split(File.separator);
if (values.length >= 2) { ... }
else if (System.getenv("FLINK_APPLICATION_ID") != null) {
    appId = System.getenv("FLINK_APPLICATION_ID");
    containerId = System.getenv("HOSTNAME");
} else { throw new RuntimeException(...); }

String pwd = System.getProperty("user.dir");
String[] segs = pwd.split(java.io.File.separatorChar == '\\' ? "\\\\" : "/");
boolean yarnLike = segs.length >= 2 && segs[segs.length-2].startsWith("application_");
if (!yarnLike) disableKafkaMetricsReporter();

try {
    reporter.open(metrics);
} catch (RuntimeException e) {
    LOG.warn("KafkaReporter cannot start (no YARN PWD): {}", e.getMessage());
    // fall back to another reporter
}

When it happens

Trigger: Running the reporter outside a YARN container (local IDE run, standalone/minikube deployment) where PWD is '/' or a shallow path, so pwd.split(File.separator).length < 2.

Common situations: Testing the job locally with the Kafka metrics reporter enabled; running on Kubernetes or a non-YARN cluster; container working directory layout changed by platform upgrade.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• Invalid host/port configuration. Host: ${host} Port: ${port}
• Could not start PrometheusReporter HTTP server on any config
• invalid elasticsearch hosts format
• invalid elasticsearch hosts format
• Unsupported client type - cannot happen

AI-assisted analysis of zhisheng17/flink-learning@d731cee761 (2026-09-06). Data as JSON: /api/errors/8fa97c5c930c405c. Report an issue: GitHub.