Re: The future of the Golang Portgroup — what to do with offline builds?

[Nils Breunese]

Hi,

I think this issue is broader than the Go ecosystem, because this goes for any software that uses a build tool that downloads dependencies at build time, like for instance Gradle and Apache Maven in the Java ecosystem, and NPM and Yarn in the JavaScript world. Either the build tool needs to have a way to provide dependencies offline, so it doesn’t fetch these dependencies at build time, and then the port maintainer needs to make sure the correct offline dependencies are provided, or we need to accept that a port uses the build tool’s dependency resolution and fetching method.

Allowing ports to download dependencies at build time carries the risk of those dependencies not being available at build time, which I guess is why MacPorts is not a fan of this method. In a corporate setting this is typically mitigated by using an in-house repository manager (e.g. Artifactory or Nexus), which caches these dependency artifacts. This is kind of similar to MacPorts servers storing files, but a repository manager provides interfaces that can be directly used by build tools to fetch dependencies, so if we’d have a MacPorts repository manager a port developer (or even better: port group) could configure a Maven/Gradle/NPM/Yarn/whatever port to use the MacPorts repository manager, and then port build should no longer depend on publicly available dependencies after the initial build.

It could be interesting to set up a repository manager for MacPorts to facilitate software that is built using build tools that download dependencies at build time, which I believe is getting more and more common in various programming ecosystems. (I’m a software developer in the Java ecosystem myself, and I’ve never used a build tool that doesn’t do this in the past 20 years or so.) But of course we’d need to have the resources (hardware and people) to set up and maintain a repository manager.

Nils.

> Op 15 feb 2024, om 04:27 heeft Austin Ziegler <halostatue at gmail.com> het volgende geschreven:
> 
> I’ve been playing with a couple of Go ports recently and trying to get various pieces to work with offline builds…and I’m not convinced that offline builds as they are currently executed are a good thing. In extreme cases, I think that the use of go.vendors is an antipattern.
> 
> As an example, I was looking at devel/please and used go2port, found a number of issues, and then applied a patch to my local golang Portgroup (all instances of them from all sources) based on [1]. Even that didn't work for everything, but I kept trying it…until I realized that I was downloading ten different versions of the same repository at ~35Mb *each* in order to build something. A build that takes about five minutes from `git clone…` to the end of `./bootstrap.sh` (devel/please is a bit strange) took more than 30 minutes just to download vendored dependencies  (many of them duplicated, remember).
> 
> The sysutils/go2port tool is amazing, but does not pull the dependencies down the way that go mod does, cannot find some dependencies because of the resolutions, and the portgroup's go.vendors support is restricted to using full repo archive downloads instead of the somewhat smarter apparently git-based solution that go mod is using where only the required subpaths are pulled.
> 
> I don't have a solution to this other than to suggest that offline builds and go.vendors be deprecated (it's also only using the legacy archive retrieval), unless we come up with another way to do this.
> 
> There might be a way to do it using `go mod vendor`, but that leaves a different problem of how / where the vendored dependencies will be stored because the upstream is not vendoring, we would be. If we had a way to take `vendor/modules.txt` (after go mod vendor) and telling one of the build machines to build vendored archives of the versioned items in the tree, we could perhaps download those, but as far as I know there is no way to *force* that particular mechanism as the archive is not coming from a source, but would be made during the initial buildbot build (until all the archived items are in place, `go mod vendor` would need to be run locally).
> 
> -a
> 
> [1] https://marc.info/?l=macports-dev&m=170710631300800&w=2
> --
> Austin Ziegler • halostatue at gmail.com <mailto:halostatue at gmail.com> • austin at halostatue.ca <mailto:austin at halostatue.ca>
> http://www.halostatue.ca/ • http://twitter.com/halostatue

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20240216/85f9e841/attachment.htm>