juanfont/headscale · warning

demultiplexing container logs: %w

Error message

demultiplexing container logs: %w

What it means

stdcopy.StdCopy failed while demultiplexing the Docker log stream into stdout and stderr buffers. Docker multiplexes ContainerLogs output with an 8-byte stdcopy frame header per write; StdCopy returns an error when the stream is truncated mid-frame, contains an invalid frame, or the underlying reader fails (network drop to the daemon).

Source

Thrown at cmd/hi/docker.go:800

		Follow:     false,
		Tail:       "all",
	})
	if err != nil {
		return fmt.Errorf("getting container logs: %w", err)
	}
	defer logReader.Close()

	// Create log files following the headscale naming convention
	stdoutPath := filepath.Join(logsDir, containerName+".stdout.log")
	stderrPath := filepath.Join(logsDir, containerName+".stderr.log")

	// Create buffers to capture stdout and stderr separately
	var stdoutBuf, stderrBuf bytes.Buffer

	// Demultiplex the Docker logs stream to separate stdout and stderr
	_, err = stdcopy.StdCopy(&stdoutBuf, &stderrBuf, logReader)
	if err != nil {
		return fmt.Errorf("demultiplexing container logs: %w", err)
	}

	// Write stdout logs
	if err := os.WriteFile(stdoutPath, stdoutBuf.Bytes(), 0o644); err != nil { //nolint:gosec,noinlineerr // log files should be readable
		return fmt.Errorf("writing stdout log: %w", err)
	}

	// Write stderr logs
	if err := os.WriteFile(stderrPath, stderrBuf.Bytes(), 0o644); err != nil { //nolint:gosec,noinlineerr // log files should be readable
		return fmt.Errorf("writing stderr log: %w", err)
	}

	if verbose {
		log.Printf("Saved logs for %s: %s, %s", containerName, stdoutPath, stderrPath)
	}

	return nil
}

View on GitHub (pinned to 565fd254d0)

Solutions

1. Re-run the test — a truncated stream from a transient daemon drop usually does not reproduce
2. Check `docker logs <container>` manually to see if the stream itself is readable or truncated
3. Prune old runs (`control_logs/` grows ~100 MB per run) and restart Docker to clear corrupted log files
4. If it reproduces deterministically for one container, inspect that container's entrypoint — a container writing raw non-framed output directly to the API socket is not the issue here, but a crashing runtime can truncate mid-frame

if _, err := stdcopy.StdCopy(&stdoutBuf, &stderrBuf, logReader); err != nil {
    if errors.Is(err, io.ErrUnexpectedEOF) {
        // truncated stream: keep whatever was demultiplexed and continue
        log.Printf("log stream truncated for %s: %v", containerName, err)
    } else {
        return fmt.Errorf("demultiplexing container logs: %w", err)
    }
}

When it happens

Trigger: The log stream from cli.ContainerLogs ends abruptly (daemon disconnect mid-read), a frame is malformed (partial read of the 8-byte header), or the reader returns a non-EOF error partway through StdCopy's loop.

Common situations: Docker daemon restarting while logs are being pulled; very large log volumes (~100 MB per run) hitting a connection reset; containers killed with SIGKILL leaving odd stream endings; inotify/filesystem issues inside the Docker VM corrupting the log file.

Related errors

• extracting logs: %w
• reading container logs: %w
• MOCKOIDC_CLIENT_ID not defined
• cleaning stale test containers: %w
• pruning networks: %w

AI-assisted analysis of juanfont/headscale@565fd254d0 (2026-08-15). Data as JSON: /api/errors/9e0af021043cdf3f. Report an issue: GitHub.