Upcoming Changes to CI [Was: Re: Time to remove Travis support?]

[Clemens Lang]

Hi everyone,

I spent some time over the last few days to update our CI environment
for base and ports. The changes in base are already merged, and I have a
PR for you below to look at the upcoming changes for ports.

## Changes that apply to base and ports

First, I've removed the Travis CI. See earlier posts in this thread on
the rationale.

I added macOS 11 and macOS 10.15 support using GitHub Actions. However,
GitHub Actions does not have a 10.14 image, so I've also made sure the
same jobs run on Azure Pipelines with a 10.14 image. This wasn't too
hard as the two systems are obviously based on the same codebase with
minor differences in configuration, output, formatting, etc.

## CI for macports-base

For builds in base, you now get build & test execution in all pull
requests and for all changes merged. I've also backported the CI changes
to the release-2.6 branch, so you'll get the same experience there.

Pushing new tags that start with "v" or pushing to the travis-ci branch
triggers automatic deployment of the tarball we use to set up the build
environment for the macports-ports CI. This was previously a manual step
during the release process, but is now automated. I have not actually
tested this with a new tag, so we'll have to keep an eye on this after
the next release.

See https://github.com/macports/macports-base/pull/220/checks for the
user experience you get for base builds now.

## CI for macports-ports

For the ports CI, I have an open pull request that showcases how I
envision the new setup:

  https://github.com/macports/macports-ports/pull/9398

Please look at that and provide feedback. I grouped the various things
we do during the CI build into steps with human-readable names,
optimized the speed of the setup and have added extensive usage of
folding groups, so that you can get a quick overview, but still dive in
where needed.

### Lint Warnings

lint warnings will now show up as a comment in the first line of the
affected Portfile. Open the "Files Changed" view and scroll to the
changed Portfiles to see them.
You may get those comments multiple times, once for each OS on GitHub
Actions, and once for every subport that's affected by this problem.
This can lead to a bit of an explosion if you trigger a lint warning on
every OS in every subport of a Portfile, but I'm hoping it won't be too
much of an issue for now (we can always improve and deduplicate later).

Unfortunately, while Azure Pipelines does support adding comments to
files, it does not seem as if those will be posted back to GitHub, so
that particular feature is 10.15 and 11 only for now.

### Build Progress & Results

I also separated dependency installation and the port's installation for
each subport into collapsible groups, so that you can quickly find the
port that failed to build for you. Note that by default, only the
progress information is shown for dependencies – I am assuming that most
people will not be interested in the details if those fail. For the
ports that where changed in this PR, you'll see the entire debugging
output.

At the end of the build (regardless of whether it failed or not), the
logfiles will be uploaded as an artifact. This includes
 - the full build log of the dependencies
 - the full stdout of the port's build
 - the main.log for the port
 - the files list and statistics we also gather on the buildbot

On Azure Pipelines, you can download the generated ZIP file from the
build results view on the summary page by following the "1 published; 1
consumed" link in the third column titled "Related" of the first row
that container the metadata of your build.

For GitHub Actions, a button with access to the artifacts will show up
at the top right on the "Checks" tab as soon as *all* builds have
finished. As long as some builds are running, you will not be able to
download the artifacts (at least I haven't found a way to do so). If
you're in a hurry to get the logs, you can always cancel the workflow,
and logs as far as available should show up.

### Bugs

The detection of changed ports needs improvement. It occasionally picks
up ports that have been merged on master since the PR was created. I
believe this happens because the default on GitHub Actions is to perform
a merge and then build the merged state, but I have not investigated the
details yet. The command I use to list the changed files is

  git diff --name-only --diff-filter=AM macports/master @

where macports/master is, of course macports/macports-ports. Maybe that
command needs to change and diff from $(git merge-base macports/master
@), or maybe we should prevent the merge from happening and build the PR
state directly, or always rebase the PR on top of
macports/macports-ports before building.

## Feedback

Please look at the current state of the PR, and leave feedback either
here via email or directly in the pull request. I'm particularly
interested in

 - whether you quickly found what you were looking for,
 - whether the output is simple to understand
 - whether the logging output is concise enough so you can quickly
   navigate, but detailed enough so you can debug your problem

See also the last few builds on
  https://github.com/macports/macports-ports/actions?query=branch%3Acal-test-ci
They may have subtle bugs that I've fixed in the latest version, but
they also built different ports and showcase the artifacts as well.

## Timeline

I think the current state is a net improvement over our current set of
Travis CI jobs that we mostly ignore because they're slow or time out,
and the currently completely broken GitHub Actions jobs, which
occasionally get canceled because the macOS 11 one fails early and stops
the 10.15 one.

Consequently I'm inclined to merge this change rather sooner than later,
i.e. probably early next week.


-- 
Clemens