apache/beam · error

failed to get manifest

Error message

failed to get manifest

What it means

Wraps a failure of GetManifest on the LegacyArtifactRetrievalService in legacyMaterialize, which is used when Materialize is called with no explicit dependencies and a legacy retrieval token. The manifest describes which artifacts exist under that token; failure means the RPC itself failed (transport or server error), with the cause wrapped under this message.

Source

Thrown at sdks/go/pkg/beam/artifact/materialize.go:299

		if _, err := w.Write(chunk.Data); err != nil {
			return "", errors.Wrapf(err, "chunk write failed")
		}
	}
	return hex.EncodeToString(sha256W.Sum(nil)), nil
}

func legacyMaterialize(ctx context.Context, endpoint string, rt string, dest string) ([]*pipepb.ArtifactInformation, error) {
	cc, err := grpcx.Dial(ctx, endpoint, 2*time.Minute)
	if err != nil {
		return nil, err
	}
	defer cc.Close()

	client := jobpb.NewLegacyArtifactRetrievalServiceClient(cc)

	m, err := client.GetManifest(ctx, &jobpb.GetManifestRequest{RetrievalToken: rt})
	if err != nil {
		return nil, errors.Wrap(err, "failed to get manifest")
	}
	mds := m.GetManifest().GetArtifact()

	var artifacts []*pipepb.ArtifactInformation
	var list []retrievable
	for _, md := range mds {
		typePayload, err := proto.Marshal(&pipepb.ArtifactFilePayload{
			Path:   md.Name,
			Sha256: md.Sha256,
		})
		if err != nil {
			return nil, errors.Wrap(err, "failed to create artifact type payload")
		}
		rolePayload, err := proto.Marshal(&pipepb.ArtifactStagingToRolePayload{
			StagedName: md.Name,
		})
		if err != nil {
			return nil, errors.Wrap(err, "failed to create artifact role payload")

View on GitHub (pinned to 12126d8942)

Solutions

1. Re-submit the pipeline to obtain a fresh retrieval token — tokens are session-scoped and expire.
2. Verify the endpoint host:port hosts the LegacyArtifactRetrievalService (correct job manager port).
3. Ensure artifacts are actually staged: pass dependencies so Materialize uses the newer ResolveArtifacts path instead of the legacy token path.
4. Check network connectivity/firewall between client and endpoint (grpcx.Dial has a 2-minute timeout).

Example fix

// before: stale token from previous submission
artifact.Materialize(ctx, endpoint, nil, "old-expired-token", dest)
// after: use fresh token from current job, or prefer dependencies
artifact.Materialize(ctx, endpoint, dependencies, freshToken, dest)

// Validate inputs before calling Materialize with legacy token path:
if len(deps) == 0 && (rt == "" || rt == artifact.NoArtifactsStaged) {
	return nil // nothing to do, skip RPC entirely
}
conn, err := net.DialTimeout("tcp", strings.TrimPrefix(endpoint, ""), 5*time.Second)
if err != nil { return fmt.Errorf("artifact endpoint unreachable: %w", err) }
conn.Close()

if err := artifact.Materialize(ctx, endpoint, nil, token, dest); err != nil {
	if strings.Contains(err.Error(), "failed to get manifest") {
		// token may be expired or endpoint wrong; refresh token then retry
		time.Sleep(5 * time.Second)
		return artifact.Materialize(ctx, endpoint, nil, freshToken(), dest)
	}
	return err
}

When it happens

Trigger: Calling artifact.Materialize with empty dependencies and a non-empty retrieval token where the GetManifest RPC fails: endpoint unreachable, retrieval token expired/unknown, or the legacy artifact service not deployed on the runner endpoint.

Common situations: Reusing an old retrieval token after the staging service discarded it; pointing at the wrong Flink/Spark job manager port; runner upgraded away from the legacy service while client code still passes a token; network/firewall blocking the gRPC port.

Related errors

• failed to retrieve chunk for %v
• failed to send chunks for %v; close error: %v
• chunk send failed
• failed to connect to data service at %v
• failed to connect to state service %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/cf6d937739904afc. Report an issue: GitHub.