[MacPorts] Meetings/MacPortsMeeting2018/BuildbotRestructuring modified

MacPorts noreply at macports.org
Tue Mar 13 14:39:55 UTC 2018


Page "Meetings/MacPortsMeeting2018/BuildbotRestructuring" was changed by mojca
Diff URL: <https://trac.macports.org/wiki/Meetings/MacPortsMeeting2018/BuildbotRestructuring?action=diff&version=2>
Revision 2
Comment: some final conclusion about the path we want to take
Changes:
-------8<------8<------8<------8<------8<------8<------8<------8<--------
Index: Meetings/MacPortsMeeting2018/BuildbotRestructuring
=========================================================================
--- Meetings/MacPortsMeeting2018/BuildbotRestructuring (version: 1)
+++ Meetings/MacPortsMeeting2018/BuildbotRestructuring (version: 2)
@@ -135,3 +135,72 @@
 }}}
 
 The approach with two separate builders is always affected by #53587 (a restart of the buildmaster will trigger pending portwatcher and portbuilder jobs at the same time), though.
+
+== Suggestions
+
+=== Features to implement
+
+* We currently lack a job which runs `portindex` (`mprsyncup`).
+* We should once run an independent job to mirror distfiles of all ports rather than having to rely on individual builds to eventually trigger mirroring of everything. Even if that job takes a week to finish. After that we could probably simplify the mirroring to potentially skip fetching files of all dependencies.
+* We should merge the "port watcher" jobs on individual builders since there is absolutely no need to build them individually. That way we can save some CPU cycles in cases where multiple commits touched the same ports (while the buildbot was busy building all other ports).
+
+=== Emails
+
+While thinking about this, buildbot was probably not designed to do the kind of builds we are running on it. It turns out that many of the ideas that we try to implement get rejected solely on the basis that such a solution would result in hard-to-follow emails and our mailing script is already overly complex.
+
+We figured out that it would probably make a lot more sense to implement our own mailer rather than trying to use ugly workarounds on the buildbot. This gives us more freedom in designing a more optimal workflow.
+
+=== Layout
+
+We propose to make the following layout schematic, either using individual builds like we did until now (but keeping just one builder per OS) or using dynamic steps from buildbot 1 and fit all the 1000 touched ports into n*1000 steps. The jobs on the global watcher would not wait for the individual builder to finish.
+
+{{{
+ +--------------+
+ |-  WATCHER   -|
+ +--------------+
+ |  (commit)    |
+ +--------------+
+ | 1-mprsyncup  |
+ +--------------+
+ | 2-mirror     |
+ +--------------+         +----------------+  +----------------+     +-----------------+
+ | 3-scheduler  |  ---->  |- 10.6 builder -|  |- 10.7 builder -| ... |- 10.13 builder -|
+ | (no waiting) |         +----------------+  +----------------+     +-----------------+
+ +--------------+         | svn up         |  | svn up         |     | svn up          |
+                          +----------------+  +----------------+     +-----------------+
+                          | clean          |  | clean          |     | clean           |
+                          +----------------+  +----------------+     +-----------------+
+                          | selfupdate     |  | selfupdate     |     | selfupdate      |
+                          +----------------+  +----------------+     +-----------------+
+                          | port list      |  | port list      |     | port list       |
+                          +----------------+  +----------------+     +-----------------+
+                          +----------------+  +----------------+     +-----------------+
+                          | dep1 install   |  | dep1 install   |     | dep1 install    |
+                          +----------------+  +----------------+     +-----------------+
+                          | dep1 archive   |  | dep1 archive   |     | dep1 archive    |
+                          +----------------+  +----------------+     +-----------------+
+                          | dep1 upload    |  | dep1 upload    |     | dep1 upload     |
+                          +----------------+  +----------------+     +-----------------+
+                          | dep1 deploy    |  | dep1 deploy    |     | dep1 deploy     |
+                          +----------------+  +----------------+     +-----------------+
+                          | clean          |  | clean          |     | clean           |
+                          +----------------+  +----------------+     +-----------------+
+                          +----------------+  +----------------+     +-----------------+
+                          | dep2 install   |  | dep2 install   |     | port3 install   |
+                          +----------------+  +----------------+     +-----------------+
+                          | dep2 archive   |  | dep2 archive   |     | port3 archive   |
+                          +----------------+  +----------------+     +-----------------+
+                          | dep2 upload    |  | dep2 upload    |     | port3 upload    |
+                          +----------------+  +----------------+     +-----------------+
+                          | dep2 deploy    |  | dep2 deploy    |     | port3 deploy    |
+                          +----------------+  +----------------+     +-----------------+
+                          +----------------+  +----------------+
+                          | port1 install  |  | port1 install  |
+                          +----------------+  +----------------+
+                          | port1 archive  |  | port1 archive  |
+                          +----------------+  +----------------+
+                          | port1 upload   |  | port1 upload   |
+                          +----------------+  +----------------+
+                          | port1 deploy   |  | port1 deploy   |
+                          +----------------+  +----------------+
+}}}
-------8<------8<------8<------8<------8<------8<------8<------8<--------

--
Page URL: <https://trac.macports.org/wiki/Meetings/MacPortsMeeting2018/BuildbotRestructuring>
MacPorts <https://www.macports.org/>
Ports system for macOS

This is an automated message. Someone added your email address to be
notified of changes on 'Meetings/MacPortsMeeting2018/BuildbotRestructuring' page.
If it was not you, please report to admin at macports.org.


More information about the macports-changes mailing list