pulumi/pulumi · critical

creating zstd encoder

Error message

creating zstd encoder: %v

What it means

This error is raised (as a panic) when the Pulumi Go SDK initializes its package-level zstd compression encoder in sdk/go/common/encoding/marshal.go. The encoder is created lazily with zstd.NewWriter(nil, zstd.WithZeroFrames(true)); because construction is expected to be infallible, any error is treated as unrecoverable and panicked. It signals a fundamental problem with the compression library setup or runtime environment rather than user data.

Solutions

  1. Verify the github.com/klauspost/compress dependency version: run `go mod tidy` and remove any replace directives or vendor corruption, then rebuild.
  2. Rebuild with a clean module cache (`go clean -modcache && go build ./...`) to rule out a corrupted build.
  3. Check available memory on the host; encoder allocation fails under OOM conditions.
  4. If reproducible, report the underlying zstd.NewWriter error text upstream, since this path is expected to be infallible.

Example fix

// before (application crashes at first compressed marshal)
out := encoding.MarshalCompressed(data) // panic: creating zstd encoder: ...
// after (pin a healthy dependency, no replaces)
// go.mod
require github.com/klauspost/compress v1.17.9
Defensive patterns

Strategy: try-catch

Try / catch

// Go panics are not catchable via errors.Is; guard at process level
func safeMarshal(data []byte) (out []byte, err error) {
  defer func() {
    if r := recover(); r != nil {
      if s, ok := r.(string); ok && strings.Contains(s, "creating zstd encoder") {
        err = fmt.Errorf("zstd init failed: %s", s)
      } else {
        panic(r)
      }
    }
  }()
  return encoding.MarshalCompressed(data)
}

Prevention

When it happens

Trigger: Calling any marshaling code path that first touches the package-level zstdEncoder sync.Once (e.g. marshaling data compressed with zstd) when zstd.NewWriter(nil, zstd.WithZeroFrames(true)) returns an error — practically only under pathological conditions such as memory allocation failure or a broken build of github.com/klauspost/compress.

Common situations: Corrupted vendor/module cache or incompatible replace directives for github.com/klauspost/compress; severe memory exhaustion at process start; running a misbuilt binary.

Related errors


AI-assisted analysis of pulumi/pulumi@793f7b2e16 (2026-08-31). Data as JSON: /api/errors/3fbd0e6b82f6cf95. Report an issue: GitHub.

Appendix: source

Thrown at sdk/go/common/encoding/marshal.go:179

	if err := reader.Close(); err != nil {
		return err
	}
	return m.inner.Unmarshal(inflated, v)
}

type zstdMarshaller struct {
	inner Marshaler
}

var (
	zstdEncoder = newZstdEncoder()
	zstdDecoder = newZstdDecoder()
)

func newZstdEncoder() *zstd.Encoder {
	enc, err := zstd.NewWriter(nil, zstd.WithZeroFrames(true))
	if err != nil {
		panic(fmt.Sprintf("creating zstd encoder: %v", err))
	}
	return enc
}

func newZstdDecoder() *zstd.Decoder {
	dec, err := zstd.NewReader(nil)
	if err != nil {
		panic(fmt.Sprintf("creating zstd decoder: %v", err))
	}
	return dec
}

func (m *zstdMarshaller) Marshal(v any) ([]byte, error) {
	b, err := m.inner.Marshal(v)
	if err != nil {
		return nil, err
	}
	return zstdEncoder.EncodeAll(b, nil), nil

View on GitHub (pinned to 793f7b2e16)