Server issues?

[Daniel J. Luke]

On Sep 8, 2017, at 1:26 PM, Ryan Schmidt <ryandesign at macports.org> wrote:
> On Sep 8, 2017, at 11:48, Daniel J. Luke wrote:
>> One solution might be to separate the build/distfile mirroring from the portfile mirroring. You could probably even do that by running separate rsync's for those on your home connection and doing some QoS setup to depref the build uploads (so they wouldn't block portfile updates).
> 
> I wouldn't know how to set up such QoS.

Most home routers have the ability to set traffic to a lower-priority queue (per application, per MAC address, per IP address, and sometimes other options). I know you can do this with OpenWRT, DD-WRT, and Netgear's router software.

Alternatively, if you have a router that supports fq_codel queuing, enabling that will probably fix it for you.

> We did originally have FAU pulling multiple rsync modules at once. This resulted in a complete clogging of my upstream bandwidth, which resulted in downstream traffic stalling as well, which resulted in builds on the buildbot failing as they could not download the needed distfiles. To prevent this, FAU now does not attempt to pull multiple modules simultaneously, and my rsync server is configured to limit its upstream bandwidth to slightly below my upstream internet speed. If we allowed 2 concurrent rsync pulls, I would have to cut the rsync bandwidth limit in half, to prevent saturating the upstream, which would make uploading binaries to the public server twice as slow as it already is.

putting the rsync traffic in a lower priority queue, or using fq_codel (which would de-pref these 'elephant' tcp streams) would fix this without having to set specific speed limits on your rsync.

>>> The buildbot setup currently consists of four Xserves and a Power Mac G5.
>> 
>> so ~ 1/4 cabinet?
> 
> I don't know. 1U for each Xserve (needs 4-post rack), plus a Power Mac G5 that is not designed to go in a rack.

it will probably be harder to find (cheap) space for non-rack-mount machine(s).

>>>>> My Internet connection is not consumer-class. If it were a consumer connection, it would be much faster and much cheaper, but ISPs does not allow running servers on consumer connections, so it is an expensive and slow business-class connection.
>>>> 
>>>> If we don't care about having static IP addresses for these machines, I have a 2 post rack in my basement that's mostly empty and a 1gig consumer connection that I'd be happy to share with the project.
>>> 
>>> Does your consumer Internet connection allow running a publicly-accessible server
>> 
>> yes, although running something like this would maybe require upgrading to a 'business' account (which they don't publish pricing for). I can ask.
> 
> Well, honestly, I'm not excited about sending my servers to someone else. And it would cost some money to ship, and there would be significant downtime while figuring out how to get everything back online at the new location. I might consider moving them to affordable colocation within driving distance of me, but I don't know where that would be. When I looked into it last year before deciding to host the servers at home, the cost to colocate was several times what I pay to have them at home.

ok, so you don't want to move the machines.

Can we move some of the work off of them? It's fine (to me) to have delays in uploading the binary archives - but it would be best if there weren't delays in getting the updated ports tree out to our end-users.

>>> We do currently have a single static IP with ports 80, 443, and 873 open for the buildbot web site and the rsync server. Although it is not entirely working at the moment, the server is also supposed to be sending emails on failed builds; my understanding is that sending mail from a dynamic address makes other servers more likely to classify the message as spam.
>> 
>> it shouldn't if it's set up correctly (ie. to relay through a smarthost).
> 
> I don't believe I currently have it set up that way. I know I experimented with it but don't think I ever got it to actually work. But if I can save the monthly cost of a static IP that would be good.

http://www.postfix.org/postconf.5.html#relayhost

it will even do smtp auth with username/password if you want/need it to.

Something like this:

relayhost = [remote_mail_hostname]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/password
smtp_sasl_security_options =
smtp_use_tls = yes
tls_random_source = dev:/dev/urandom


-- 
Daniel J. Luke