<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[154434] contrib/buildbot-test</title>
</head>
<body>
<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; }
#msg dl a { font-weight: bold}
#msg dl a:link { color:#fc3; }
#msg dl a:active { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { white-space: pre-line; overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="https://trac.macports.org/changeset/154434">154434</a></dd>
<dt>Author</dt> <dd>larryv@macports.org</dd>
<dt>Date</dt> <dd>2016-10-29 07:00:03 +0200 (Sat, 29 Oct 2016)</dd>
</dl>
<h3>Log Message</h3>
<pre>buildbot: Switch to GitHub
The production setup will now retrieve all source code from GitHub.
- Tweak `port_from_path` to account for loss of the "dports" prefix.
- Add new configuration keys for the GitHub webhook secret and the URLs
of each Git repository. The URLs must end with ".git", for now.
- Receive webhook payloads by enabling the GitHub change hook on the
WebStatus status target. The hook requires both basic authentication
and a secret. The credentials must be stored in plaintext in the
htpasswd file; the secret, in config.json.
- Now that each MacPorts component exists in its own repository, filter
changes based on their "repository" properties instead of vetting the
leading component of changed files' paths. And only allow changes from
"master" branches.
- Fetch base, mpbb, the website, and the guide using Git. Allow `mpbb
checkout` to default to Git by dropping `--svn-url`.
- Stop doing link transformations in the web interface: The GitHub
change hook automatically adds a "revlink" attribute to changes, and
we'll be using full Trac URLs in commit messages. (We may restore
transformations in the future to link "#123" to GitHub pull requests.)</pre>
<h3>Modified Paths</h3>
<ul>
<li><a href="#contribbuildbottestconfigjsonsample">contrib/buildbot-test/config.json.sample</a></li>
<li><a href="#contribbuildbottestmastercfg">contrib/buildbot-test/master.cfg</a></li>
</ul>
</div>
<div id="patch">
<h3>Diff</h3>
<a id="contribbuildbottestconfigjsonsample"></a>
<div class="modfile"><h4>Modified: contrib/buildbot-test/config.json.sample (154433 => 154434)</h4>
<pre class="diff"><span>
<span class="info">--- contrib/buildbot-test/config.json.sample 2016-10-29 01:54:38 UTC (rev 154433)
+++ contrib/buildbot-test/config.json.sample 2016-10-29 05:00:03 UTC (rev 154434)
</span><span class="lines">@@ -2,6 +2,7 @@
</span><span class="cx"> "production": false,
</span><span class="cx"> "privkey": "/var/keys/macports-privkey.pem",
</span><span class="cx"> "buildboturl": "http://domain.tld:8010/",
</span><ins>+ "githubsecret": "",
</ins><span class="cx"> "slaveprefix": "/opt/local",
</span><span class="cx"> "toolsprefix": "/opt/mports",
</span><span class="cx"> "deploy": {
</span></span></pre></div>
<a id="contribbuildbottestmastercfg"></a>
<div class="modfile"><h4>Modified: contrib/buildbot-test/master.cfg (154433 => 154434)</h4>
<pre class="diff"><span>
<span class="info">--- contrib/buildbot-test/master.cfg 2016-10-29 01:54:38 UTC (rev 154433)
+++ contrib/buildbot-test/master.cfg 2016-10-29 05:00:03 UTC (rev 154434)
</span><span class="lines">@@ -32,9 +32,8 @@
</span><span class="cx"> def port_from_path(path, sep='/'):
</span><span class="cx"> components = path.split(sep)
</span><span class="cx"> try:
</span><del>- if (components[0] == 'dports' and components[1] != '_resources'
- and components[3] in ('Portfile', 'files')):
- return components[2]
</del><ins>+ if (components[0] != '_resources' and components[2] in ('Portfile', 'files')):
+ return components[1]
</ins><span class="cx"> except IndexError:
</span><span class="cx"> pass
</span><span class="cx"> # Might be better to throw a custom exception here?
</span><span class="lines">@@ -58,9 +57,13 @@
</span><span class="cx"> 'workersfile': _path('slaves.json'),
</span><span class="cx"> 'htpasswdfile': _path('htpasswd'),
</span><span class="cx">
</span><del>- # Sources
- 'mpbbsvnurl': 'https://svn.macports.org/repository/macports/contrib/mp-buildbot',
- 'svnurl': 'https://svn.macports.org/repository/macports/trunk',
</del><ins>+ # GitHub. Repository URLs must have the ".git" suffix.
+ 'githubsecret': '',
+ 'baseurl': 'https://github.com/macports/macports-base.git',
+ 'mpbburl': 'https://github.com/macports/mpbb.git',
+ 'portsurl': 'https://github.com/macports/macports-ports.git',
+ 'wwwurl': 'https://github.com/macports/macports-www.git',
+ 'guideurl': 'https://github.com/macports/macports-guide.git',
</ins><span class="cx">
</span><span class="cx"> # Tooling
</span><span class="cx"> 'slaveprefix': '/opt/local',
</span><span class="lines">@@ -126,49 +129,53 @@
</span><span class="cx"> c['slavePortnum'] = config['slaveport']
</span><span class="cx">
</span><span class="cx">
</span><del>-####### CHANGESOURCES #######
</del><ins>+####### WEB STATUS AND CHANGE HOOKS #######
</ins><span class="cx">
</span><del>-# the 'change_source' setting tells the buildmaster how it should find out
-# about source code changes.
</del><ins>+# WebStatus [1] runs a web server that serves the web interface. It can
+# also accept HTTP requests at a hook endpoint [2] and translate them
+# into Changes. We're using the GitHub hook [3] to accept webhook
+# payloads [4].
+#
+# [1]: http://docs.buildbot.net/0.8.12/manual/cfg-statustargets.html#webstatus
+# [2]: http://docs.buildbot.net/0.8.12/manual/cfg-statustargets.html#change-hooks
+# [3]: http://docs.buildbot.net/0.8.12/manual/cfg-statustargets.html#github-hook
+# [4]: https://developer.github.com/webhooks
</ins><span class="cx">
</span><del>-# poller is used for local testing but PBChangeSource (which relies on
-# notifications from a post-commit script) should be used in production
-
</del><span class="cx"> if config['production']:
</span><del>- # TODO
- sourcedata = []
-# with open(_path('source.json')) as f:
-# sourcedata = json.load(f)
-# c['change_source'] = changes.PBChangeSource(user=sourcedata[0], passwd=sourcedata[1], port=sourcedata[2])
</del><ins>+ # The username doesn't correspond to any real user and can be set to
+ # an arbitrary value. The password must be stored in plaintext and
+ # can be added to the password file with "htpasswd -p".
+ change_hook_kwargs = {
+ 'change_hook_auth': ['file:' + config['htpasswdfile']],
+ 'change_hook_dialects': {
+ 'github': {
+ 'secret': config['githubsecret'].encode('utf-8'),
+ 'strict': True
+ }
+ }
+ }
</ins><span class="cx"> else:
</span><del>- c['change_source'] = changes.SVNPoller(
- svnurl='https://svn.macports.org/repository/macports/trunk',
- #svnbin='/opt/local/bin/svn',
- pollinterval=300,
- category='macports',
- project='ports')
</del><ins>+ # TODO Add alternate change source, probably a GitPoller.
+ change_hook_kwargs = {}
</ins><span class="cx">
</span><ins>+c['status'] = [
+ WebStatus(
+ http_port=config['httpport'],
+ authz=util.Authz(
+ auth=util.HTPasswdAprAuth(config['htpasswdfile']),
+ gracefulShutdown='auth',
+ forceBuild='auth',
+ forceAllBuilds='auth',
+ pingBuilder='auth',
+ stopBuild='auth',
+ stopAllBuilds='auth',
+ cancelPendingBuild='auth'),
+ **change_hook_kwargs)
+ ]
</ins><span class="cx">
</span><ins>+
</ins><span class="cx"> ####### SCHEDULERS #######
</span><span class="cx">
</span><del>-def change_has_base(change):
- for f in change.files:
- if f.startswith('base'):
- return True
- return False
-
-def change_has_guide(change):
- for f in change.files:
- if f.startswith('doc-new'):
- return True
- return False
-
-def change_has_www(change):
- for f in change.files:
- if f.startswith('www'):
- return True
- return False
-
</del><span class="cx"> base_platforms = [plat for plat in build_platforms if 'legacy' not in plat and '10.6_i386' not in plat]
</span><span class="cx"> port_platforms = [plat for plat in build_platforms if 'linux' not in plat and '10.5_ppc' != plat]
</span><span class="cx">
</span><span class="lines">@@ -177,17 +184,21 @@
</span><span class="cx"> portbuilder_buildernames = map('ports-{}-builder'.format, port_platforms)
</span><span class="cx"> portbuilder_triggerables = map('ports-{}-trigger'.format, port_platforms)
</span><span class="cx">
</span><ins>+# The ChangeFilters assume that Git URLs end with ".git".
</ins><span class="cx"> c['schedulers'] = [
</span><span class="cx"> schedulers.SingleBranchScheduler(
</span><span class="cx"> name='base',
</span><span class="cx"> treeStableTimer=None,
</span><span class="cx"> change_filter=util.ChangeFilter(
</span><del>- filter_fn=change_has_base),
</del><ins>+ repository=config['baseurl'][:-4],
+ branch='master'),
</ins><span class="cx"> builderNames=base_buildernames),
</span><span class="cx"> schedulers.SingleBranchScheduler(
</span><span class="cx"> name='ports',
</span><span class="cx"> treeStableTimer=None,
</span><span class="cx"> change_filter=util.ChangeFilter(
</span><ins>+ repository=config['portsurl'][:-4],
+ branch='master',
</ins><span class="cx"> # Should actually skip changes to files/ only, but only if
</span><span class="cx"> # we know the last build of the port succeeded.
</span><span class="cx"> filter_fn=lambda change: any(port_from_path(f) for f in change.files)),
</span><span class="lines">@@ -222,7 +233,8 @@
</span><span class="cx"> name='www',
</span><span class="cx"> treeStableTimer=300,
</span><span class="cx"> change_filter=util.ChangeFilter(
</span><del>- filter_fn=change_has_www),
</del><ins>+ repository=config['wwwurl'][:-4],
+ branch='master'),
</ins><span class="cx"> builderNames=['docs-www']),
</span><span class="cx"> schedulers.ForceScheduler(
</span><span class="cx"> name='www_force',
</span><span class="lines">@@ -235,7 +247,8 @@
</span><span class="cx"> name='guide',
</span><span class="cx"> treeStableTimer=300,
</span><span class="cx"> change_filter=util.ChangeFilter(
</span><del>- filter_fn=change_has_guide),
</del><ins>+ repository=config['guideurl'][:-4],
+ branch='master'),
</ins><span class="cx"> builderNames=['docs-guide']),
</span><span class="cx"> schedulers.ForceScheduler(
</span><span class="cx"> name='guide_force',
</span><span class="lines">@@ -259,10 +272,11 @@
</span><span class="cx"> base_factory = util.BuildFactory()
</span><span class="cx"> base_factory.workdir = '../build'
</span><span class="cx">
</span><del>-#base_factory.addStep(steps.SVN(
-# repourl=util.Interpolate('https://svn.macports.org/repository/macports/%(src::branch:-trunk)s/base'),
-base_factory.addStep(steps.SVN(
- repourl='https://svn.macports.org/repository/macports/trunk/base',
</del><ins>+# Set progress=True on Git steps to prevent timeouts on slow fetches.
+base_factory.addStep(steps.Git(
+ repourl=config['baseurl'],
+ progress=True,
+ mode='full',
</ins><span class="cx"> method='copy',
</span><span class="cx"> env={'PATH': path_ports}))
</span><span class="cx"> base_factory.addStep(steps.Configure(command=util.WithProperties("""
</span><span class="lines">@@ -297,7 +311,7 @@
</span><span class="cx"> # support forced build properties
</span><span class="cx"> ports = set(self.getProperty('portlist', default='').split())
</span><span class="cx">
</span><del>- # paths should be dports/category/portdir(/...)
</del><ins>+ # paths should be category/portdir(/...)
</ins><span class="cx"> ports.update(ifilter(None, imap(port_from_path, self.build.allFiles())))
</span><span class="cx">
</span><span class="cx"> self.setProperty('fullportlist', ' '.join(ports))
</span><span class="lines">@@ -350,13 +364,12 @@
</span><span class="cx"> portwatcher_factory.useProgress = False
</span><span class="cx"> portwatcher_factory.workdir = '../build'
</span><span class="cx">
</span><del>- # get mp-buildbot; we'll do the checkout of base and dports via these scripts
- portwatcher_factory.addStep(steps.SVN(
- repourl=config['mpbbsvnurl'],
</del><ins>+ # get mpbb; we'll do the checkout of base and dports via these scripts
+ portwatcher_factory.addStep(steps.Git(
+ repourl=config['mpbburl'],
+ progress=True,
</ins><span class="cx"> env={'PATH': path_ports},
</span><span class="cx"> alwaysUseLatest=True,
</span><del>- preferLastChangedRev=True,
- mode='incremental',
</del><span class="cx"> workdir=os.path.join(portwatcher_factory.workdir, 'mpbb'),
</span><span class="cx"> haltOnFailure=True))
</span><span class="cx">
</span><span class="lines">@@ -374,7 +387,7 @@
</span><span class="cx"> haltOnFailure=True))
</span><span class="cx">
</span><span class="cx"> portwatcher_factory.addStep(steps.ShellCommand(
</span><del>- command=['./mpbb/mpbb', '--prefix', util.WithProperties(prefix), 'checkout', '--svn-url', config['svnurl']],
</del><ins>+ command=['./mpbb/mpbb', '--prefix', util.WithProperties(prefix), 'checkout', '--ports-url', config['portsurl']],
</ins><span class="cx"> timeout=3600,
</span><span class="cx"> name='checkout',
</span><span class="cx"> description=['syncing', 'ports'],
</span><span class="lines">@@ -502,8 +515,9 @@
</span><span class="cx"> if 'www' in config['deploy']:
</span><span class="cx"> docs_www_factory = util.BuildFactory()
</span><span class="cx"> # TODO: incremental mode with cleanup?
</span><del>- docs_www_factory.addStep(steps.SVN(
- repourl='https://svn.macports.org/repository/macports/trunk/www',
</del><ins>+ docs_www_factory.addStep(steps.Git(
+ repourl=config['wwwurl'],
+ progress=True,
</ins><span class="cx"> mode='full',
</span><span class="cx"> method='copy',
</span><span class="cx"> workdir='www'))
</span><span class="lines">@@ -520,8 +534,9 @@
</span><span class="cx"> if 'guide' in config['deploy']:
</span><span class="cx"> docs_guide_factory = util.BuildFactory()
</span><span class="cx"> # TODO: incremental mode with cleanup?
</span><del>- docs_guide_factory.addStep(steps.SVN(
- repourl='https://svn.macports.org/repository/macports/trunk/doc-new',
</del><ins>+ docs_guide_factory.addStep(steps.Git(
+ repourl=config['guideurl'],
+ progress=True,
</ins><span class="cx"> mode='full',
</span><span class="cx"> method='copy',
</span><span class="cx"> workdir='guide'))
</span><span class="lines">@@ -608,26 +623,8 @@
</span><span class="cx"> env=merge_dicts(env_buildinfo, {'PATH': path_ports})))
</span><span class="cx">
</span><span class="cx">
</span><del>-####### STATUS TARGETS #######
</del><ins>+####### MAIL NOTIFIERS #######
</ins><span class="cx">
</span><del>-# 'status' is a list of Status Targets. The results of each build will be
-# pushed to these targets. buildbot/status/*.py has a variety to choose from,
-# including web pages, email senders, and IRC bots.
-
-c['status'] = []
-
-htauth = util.HTPasswdAprAuth(config['htpasswdfile'])
-
-authz_cfg = util.Authz(
- auth=htauth,
- gracefulShutdown='auth',
- forceBuild='auth',
- forceAllBuilds='auth',
- pingBuilder='auth',
- stopBuild='auth',
- stopAllBuilds='auth',
- cancelPendingBuild='auth')
-
</del><span class="cx"> # TODO: This is the old mail notifier;
</span><span class="cx"> # - useful functionality could be copied
</span><span class="cx"> # - then the code should be removed
</span><span class="lines">@@ -805,15 +802,8 @@
</span><span class="cx"> c['title'] = config['title']
</span><span class="cx"> c['titleURL'] = config['titleurl']
</span><span class="cx"> c['buildbotURL'] = config['buildboturl']
</span><del>-c['status'].append(WebStatus(
- http_port=config['httpport'],
- authz=authz_cfg,
- changecommentlink=(r'#(\d+)', r'https://trac.macports.org/ticket/\1', r'Ticket \g<0>')))
</del><span class="cx">
</span><del>-c['revlink'] = util.RevlinkMatch([r'https://svn.macports.org/repository/macports/(.*)'],
- r'https://trac.macports.org/changeset/%s')
</del><span class="cx">
</span><del>-
</del><span class="cx"> ####### DATABASE #######
</span><span class="cx">
</span><span class="cx"> # This specifies what database buildbot uses to store its state. You can
</span></span></pre>
</div>
</div>
</body>
</html>