Re: The future of the Golang Portgroup ― what to do with offline builds?

[Austin Ziegler]

For some dependency management systems, trust is implied and not verified.
The MacPorts approach can definitely provide benefits here: a dependency is
declared with checksums, and it is compared against the downloaded version.
This can happen with Ruby gems, Cargo crates, Python packages, NPM, etc.
Where there is a single distribution arm (even if mirrored like MacPorts
itself), and presumably for some "distributed" distribution systems (Maven?
I don't do Java), then MacPorts provides benefits. In most cases, these
dependencies are *packages*, which means that they are fancy tarballs of a
subset of the whole development tree.

This is not the case for Go, where `go.sum` stores the digest of the
dependencies and the "distribution" system is git. Like the Node community,
there is a common pattern (especially with large API providers like AWS and
Google) of using monorepos. There are ways to use git to only clone
specific subfolders (git clone --filter and git sparse-checkout) at tags,
and while I cannot guarantee that this is what Go uses, it certainly does
not do what the Golang portgroup is doing when you use go.vendors.

A new release of the whole AWS SDK v2 for Go was cut yesterday, but that's
not how the go dependencies are computed. If I just need S3 and ACM, I
would use `import "github.com/aws/aws-sdk-go-v2/service/s3"` and `import "
github.com/aws/aws-sdk-go-v2/service/acm"`. I might do `go get
github.com/aws/aws-sdk-go-v2/service/s3 at v1.49.0` and `go get
github.com/aws/aws-sdk-go-v2/acm at v1.23.1` (those are both the latest
versions of the individual service SDKs).

For go.vendors to work, I have to download the WHOLE repository at
`service/s3/v1.49.0` and `service/acm/v1.23.1` tags, which is dozens of
times larger than what `go get` pulls down (almost 70Mb each). `go get` on
ACM downloads about 7MB of code and dependencies.  If I use ten different
AWS dependencies, each MacPorts build machine (and the developer trying to
update `go.vendors`) has to download nearly 700Mb of dependencies instead
of 50–100Mb (go get will not download dependencies already in its cache)
for those ten dependencies.

This is why I focused exclusively on the Golang portgroup. I don't think
that the MacPorts model is fundamentally broken. I appreciate it. But I
think that go.vendors is the worst approach for Go, and I think that it
makes things actively worse from a port and dependency management
perspective because it adds little over Go's existing online mechanisms. If
Go mirrors could be treated as targets for `GOPROXY`, then I think we would
see substantial benefit. There's a bootstrapping question on this, for how
a dependency would make it onto the mirrors in the first place, but I’m
raising this to try to get possible ideas for the solution.

-a

On Fri, Feb 16, 2024 at 9:58 AM Kirill A. Korinsky <kirill at korins.ky> wrote:

> On Fri, 16 Feb 2024 10:24:59 +0100,
> Nils Breunese wrote:
> >
> > Allowing ports to download dependencies at build time carries the risk of
> > those dependencies not being available at build time, which I guess is
> why
> > MacPorts is not a fan of this method. In a corporate setting this is
> typically
> > mitigated by using an in-house repository manager (e.g. Artifactory or
> Nexus),
> > which caches these dependency artifacts. This is kind of similar to
> MacPorts
> > servers storing files, but a repository manager provides interfaces that
> can
> > be directly used by build tools to fetch dependencies, so if we’d have a
> > MacPorts repository manager a port developer (or even better: port group)
> > could configure a Maven/Gradle/NPM/Yarn/whatever port to use the MacPorts
> > repository manager, and then port build should no longer depend on
> publicly
> > available dependencies after the initial build.
> >
>
> Let switch to another and usually missed aspect of automatic downloading
> dependency from interten to build a software: security.
>
> For example Maven as the most popular Java build system doesn't care about
> checksums for dependency.
>
> It ultimatley trust used repository and when it builds software it can
> inject
> any unexpected code.
>
> From another side, MacPorts enforces checksuming of dependency that
> garantue
> that it is the same artifact that was used before.
>
> --
> wbr, Kirill
>


-- 
Austin Ziegler • halostatue at gmail.com • austin at halostatue.ca
http://www.halostatue.ca/ • http://twitter.com/halostatue
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20240216/f8b0ae1b/attachment.htm>