Volunteer for a workshop on "setting up your own buildbot/buildslave"?

[Ryan Schmidt]

On Nov 11, 2015, at 8:25 AM, Rainer Müller wrote:

> On 2015-11-11 14:18, Ryan Schmidt wrote:
>>> You usually do not want to fetch archives for
>>> anything else than the current OS,
>> 
>> Of course we would not want that.
>> 
>>> so it is implicit.
>> 
>> I don't know what you mean by this.
> 
> I meant there are no settings in archive_sites.tcl or
> archive_sites.conf, but OS and OS release are still considered when
> fetching an archive.


>>> We could also host the packages in subdirectories:
>>> .../${os.platform}/${os.major}/${build_arch}/*.tbz2
>>> However, you would have to replicate this structure on fetch. It would
>>> not make much difference whether this is in the path or in the filename.
>> 
>> Yes, for my hypothetical scenario, I was assuming that, in the absence of the OS name and version being in the filename, it would be in a directory name.
>> 
>> packages.m.o/darwin_15/boost/...
>> packages.m.o/darwin_10/boost/...
>> packages.m.o/darwin_10-libc++/boost/...
> 
> We would then need two entries for these archive sites in
> _resources/port1.0/fetch/archive_sites.tcl in the ports tree. For darwin
>> = 13 the default of cxx_stdlib changed. Would we just make the
> definition there switch on the OS release?
> 
> Also note that this would need to be reflected locally. Currently port
> archives are stored at ${prefix}/var/macports/software/${name}/*.tbz2.
> If we remove OS release from the filename, they need to go into the
> corresponding subdirectories instead. This path is also stored in the
> registry and would need to be replaced with a base upgrade.

Ok, so those are more reasons against doing such a reorganization. Sounds like putting it in the filename is simpler to implement.


>> Having all of a port's packages -- for all platforms, variants and
>> build options -- in a single directory is nice though because it's a
>> single directory listing to look at to figure out if a particular
>> binary exists. I use this all the time when deciding whether I need
>> to force a buildbot build or investigate a buildbot build failure,
>> and it would be less good for there to be even two directories to
>> need to check, much less more than that.
> 
> I see your point here.
> 
> This could also be solved with a proper new web interface for the
> PortIndex, that regularly checks which archives are available. However,
> from experience with this topic, either nobody will write this or we
> will fail once again to deploy it...
> 
>>> Is your intention to build all packages twice, for both libc++ and
>>> libstdc++? Is that even worth the effort?
>> 
>> Yes, my intention is that there would be additional buildbot slave
>> servers set up for libc++ on each older OS version. Why wouldn't that
>> be worth the effort? Are you suggesting as an alternative that we
>> would immediately switch the older OS buildbot slaves over to libc++
>> and no longer provide binaries for libstdc++? That would mean we
>> would need an immediate plan for how to switch users from libstdc++
>> to libc++. I was hoping to take this one problem at a time, not all
>> problems at once.
> 
> Is there demand for archives linked against libc++? If that is high
> enough, it should become our default and we would not need the archives
> linked against libstdc++ anymore.

There has been repeated demand for a libc++ buildbot for older systems, which cannot be done until MacPorts base can differentiate libc++ archives from libstdc++ archives.


> First setting up another buildbot instance with another configuration
> makes sense on the way to this. But if we do not intent to switch the
> default later, I do not see a need for this.

I think the order of operations would be:

- Figure out a way to have libstdc++ and libc++ archives coexist on the packages server by either modifying the archive filename or by setting up a second packages server (or subdirectory of the existing server, which would be simpler for mirrors)
- Get new buildbot slaves to build libc++ archives, which will get us build reports about what's broken so that we can fix it
- Some developers or user switch to libc++ and test more
- Develop a strategy for how to migrate libstdc++ users to libc++
- Make libc++ the default and transition users to it

There might be users who want to stay with libstdc++ because they use MacPorts-built libraries for purposes outside of MacPorts. They could do that by deliberately setting cxx_stdlib to libstdc++ in macports.conf. No default value is present for it, so users who haven't specified it would get upgraded to our new default.

I guess most users just want to use ports installed by MacPorts, and would be well served by MacPorts switching to libc++, since more and more ports are unable to build with libstdc++.