<pre style='margin:0'>
Ryan Schmidt (ryandesign) pushed a commit to branch master
in repository macports-infrastructure.
</pre>
<p><a href="https://github.com/macports/macports-infrastructure/commit/74c01e8f6db37295933101610384abf974f4cddb">https://github.com/macports/macports-infrastructure/commit/74c01e8f6db37295933101610384abf974f4cddb</a></p>
<pre style="white-space: pre; background: #F8F8F8">The following commit(s) were added to refs/heads/master by this push:
<span style='display:block; white-space:pre;color:#404040;'> new 74c01e8 mprsyncup: Improve generation and deployment of tarballs and portindexes
</span>74c01e8 is described below
<span style='display:block; white-space:pre;color:#808000;'>commit 74c01e8f6db37295933101610384abf974f4cddb
</span>Author: Ryan Schmidt <ryandesign@macports.org>
AuthorDate: Wed Nov 16 02:45:21 2016 -0600
<span style='display:block; white-space:pre;color:#404040;'> mprsyncup: Improve generation and deployment of tarballs and portindexes
</span><span style='display:block; white-space:pre;color:#404040;'>
</span><span style='display:block; white-space:pre;color:#404040;'> Only generate and sign base.tar if base changed.
</span><span style='display:block; white-space:pre;color:#404040;'>
</span><span style='display:block; white-space:pre;color:#404040;'> Only generate and sign ports.tar and portindexes if ports changed.
</span><span style='display:block; white-space:pre;color:#404040;'>
</span><span style='display:block; white-space:pre;color:#404040;'> Incorporate functionality of porttree_tarball.sh to generate
</span><span style='display:block; white-space:pre;color:#404040;'> ports.tar.gz if ports changed.
</span><span style='display:block; white-space:pre;color:#404040;'>
</span><span style='display:block; white-space:pre;color:#404040;'> No longer try to update files in a temporary directory. It wasn’t
</span><span style='display:block; white-space:pre;color:#404040;'> atomic and was causing more problems than it solved, presumably
</span><span style='display:block; white-space:pre;color:#404040;'> including: https://trac.macports.org/ticket/52629
</span><span style='display:block; white-space:pre;color:#404040;'>
</span><span style='display:block; white-space:pre;color:#404040;'> The new goal is to reduce the amount of time that the files on the
</span><span style='display:block; white-space:pre;color:#404040;'> rsync server are in an inconsistent state. Use GNU mv to atomically
</span><span style='display:block; white-space:pre;color:#404040;'> replace individual files.
</span><span style='display:block; white-space:pre;color:#404040;'>
</span><span style='display:block; white-space:pre;color:#404040;'> Use hardlinks because it's faster than copying, reduces server disk
</span><span style='display:block; white-space:pre;color:#404040;'> space, and hopefully reduces rsync transfer time.
</span>---
jobs/mprsyncup | 68 +++++++++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 53 insertions(+), 15 deletions(-)
<span style='display:block; white-space:pre;color:#808080;'>diff --git a/jobs/mprsyncup b/jobs/mprsyncup
</span><span style='display:block; white-space:pre;color:#808080;'>index 780e537..d3a88a5 100755
</span><span style='display:block; white-space:pre;background:#e0e0ff;'>--- a/jobs/mprsyncup
</span><span style='display:block; white-space:pre;background:#e0e0ff;'>+++ b/jobs/mprsyncup
</span><span style='display:block; white-space:pre;background:#e0e0e0;'>@@ -36,10 +36,12 @@ SVN="/opt/local/bin/svn --non-interactive"
</span> RSYNC="/opt/local/bin/rsync -q"
RM="/bin/rm"
MKDIR="/bin/mkdir"
<span style='display:block; white-space:pre;background:#e0ffe0;'>+MV="/opt/local/bin/gmv"
</span> LN="/bin/ln"
TAR="/usr/bin/tar"
OPENSSL="/usr/bin/openssl"
SED="/usr/bin/sed"
<span style='display:block; white-space:pre;background:#e0ffe0;'>+STAT="/opt/local/bin/gstat"
</span>
# Paths we'll work on:
ROOT=/var/tmp/macports
<span style='display:block; white-space:pre;background:#e0e0e0;'>@@ -69,6 +71,16 @@ PRIVKEY=""
</span> # Functions
#
<span style='display:block; white-space:pre;background:#e0ffe0;'>+hardlink() {
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ SOURCE="$1"
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ TARGET="$2"
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ [ -f "${SOURCE}" ] && [ -f "${TARGET}" ] && [ "$(${STAT} -c %i "${SOURCE}")" = "$(${STAT} -c %i "${TARGET}")" ] && return
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ printf "Hard linking %s to %s\n" "${SOURCE}" "${TARGET}"
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ TMPTARGET="${ROOT}"/"${TARGET##*/}".$$
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${LN} "${SOURCE}" "${TMPTARGET}"
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${MV} "${TMPTARGET}" "${TARGET}"
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+}
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+
</span> sign() {
[ -z "${PRIVKEY}" ] && return
for FILE in "$@"; do
<span style='display:block; white-space:pre;background:#e0e0e0;'>@@ -186,18 +198,44 @@ fi
</span> # Update release/tarballs
#
<span style='display:block; white-space:pre;background:#ffe0e0;'>-# generate and sign tarballs of base and dports
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-# the signature always needs to match, so we try to make this look atomic to
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-# clients by switching a symlink target
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-TAR_CURDIR=${RSYNCROOT}/release/tarballs_current
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-${MKDIR} -p ${TAR_CURDIR}
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-cp -pR ${TAR_CURDIR} ${RSYNCROOT}/release/tarballs_old
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-${LN} -sfh tarballs_old ${RSYNCROOT}/release/tarballs
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-${TAR} -C ${RSYNCROOT}/release/ -cf ${TAR_CURDIR}/base.tar base
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-${TAR} --exclude 'PortIndex*' -C ${RSYNCROOT}/release/ -cf ${TAR_CURDIR}/ports.tar ports
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-cp -pR ${RSYNCROOT}/release/ports/PortIndex_* ${TAR_CURDIR}
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-sign ${TAR_CURDIR}/base.tar
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-sign ${TAR_CURDIR}/ports.tar ${TAR_CURDIR}/PortIndex_*/PortIndex
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-${LN} -sfh tarballs_current ${RSYNCROOT}/release/tarballs
</span><span style='display:block; white-space:pre;background:#ffe0e0;'>-${RM} -rf ${RSYNCROOT}/release/tarballs_old
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+# Generate and sign tarballs of base and ports and the PortIndex files.
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+if [ ${RBASE_CHANGED} -eq 1 ]; then
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${TAR} -C "${RSYNCROOT}"/release/ -cf "${ROOT}"/base.tar base
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ sign "${ROOT}"/base.tar
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+fi
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+if [ ${PORTS_CHANGED} -eq 1 ]; then
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${TAR} -C "${RSYNCROOT}"/release/ -czf "${ROOT}"/ports.tar.gz ports
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${TAR} --exclude 'PortIndex*' -C "${RSYNCROOT}"/release/ -cf "${ROOT}"/ports.tar ports
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ for INDEX_DIR in "${RSYNCROOT}"/release/ports/PortIndex_*; do
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ INDEX_LINK_DIR="${ROOT}"/"${INDEX_DIR##*/}"
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${MKDIR} -p "${INDEX_LINK_DIR}"
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ hardlink "${INDEX_DIR}"/PortIndex "${INDEX_LINK_DIR}"/PortIndex
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ done
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ sign "${ROOT}"/ports.tar "${ROOT}"/PortIndex_*/PortIndex
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+fi
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+# Tarballs used to be a symlink to the real directory tarballs_current.
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+# Clean up this situation if found. This should only happen once.
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+[ -L "${RSYNCROOT}"/release/tarballs ] && ${RM} -f "${RSYNCROOT}"/release/tarballs
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+[ -d "${RSYNCROOT}"/release/tarballs_current ] && ${MV} "${RSYNCROOT}"/release/tarballs_current "${RSYNCROOT}"/release/tarballs
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+# Replace files on rsync server as quickly as possible.
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+# This is not atomic but doing it atomically is difficult.
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+${MKDIR} -p "${RSYNCROOT}"/release/tarballs "${RSYNCROOT}"/distfiles
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+if [ ${RBASE_CHANGED} -eq 1 ]; then
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${MV} "${ROOT}"/base.tar* "${RSYNCROOT}"/release/tarballs
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+fi
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+if [ ${PORTS_CHANGED} -eq 1 ]; then
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${MV} "${ROOT}"/ports.tar.gz "${RSYNCROOT}"/release
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ hardlink "${RSYNCROOT}"/release/ports.tar.gz "${RSYNCROOT}"/distfiles/ports.tar.gz
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${MV} "${ROOT}"/ports.tar* "${RSYNCROOT}"/release/tarballs
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ for INDEX_DIR in "${RSYNCROOT}"/release/ports/PortIndex_*; do
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ INDEX_LINK_DIR="${RSYNCROOT}"/release/tarballs/${INDEX_DIR##*/}
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${MKDIR} -p "${INDEX_LINK_DIR}"
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ hardlink "${INDEX_DIR}"/PortIndex "${INDEX_LINK_DIR}"/PortIndex
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ hardlink "${INDEX_DIR}"/PortIndex.quick "${INDEX_LINK_DIR}"/PortIndex.quick
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ hardlink "${ROOT}"/"${INDEX_DIR##*/}"/PortIndex.rmd160 "${INDEX_LINK_DIR}"/PortIndex.rmd160
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ done
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+ ${RM} -rf "${ROOT}"/PortIndex_*
</span><span style='display:block; white-space:pre;background:#e0ffe0;'>+fi
</span></pre><pre style='margin:0'>
</pre>