macports fetch suggestion

[Daniel Price]

On 12 Sep 2011, at 16:40, Ryan Schmidt wrote:

> 
> On Sep 12, 2011, at 01:13, Daniel Price wrote:
> 
>> Thanks for the tip -- the "port fetch rdepof" is just what I was looking for.
>> 
>> Is there any particular reason the prefetching cannot be the default though?
> 
> What's your suggestion: that MacPorts would just run "sudo port fetch $name rdepof:$name" automatically every time before trying to install a port? Computing dependencies can be slow, as can fetching; users who have already installed some dependencies and then deliberately deleted the no-longer-needed distfiles wouldn't want to have them re-fetched. Also, any ports that fetch from a version control system instead of from a tarball currently have to re-fetch every time; there is no local caching for that. So this would incur unnecessary downloads and delays.
> 
> You might then say we should only pre-fetch ports that are not already installed and up to date. Ok, we could run "sudo port fetch $name rdepof:$name and \( outdated or not installed \)". But computing lists of installed and outdated ports takes time too. Running that command on my system on the kdegames4 port took 45 seconds; I don't really want all my installs to be slowed down by that much to pre-fetch files. Granted it would be faster running before an install, since the install must already compute what's installed and outdated.
> 
> There's also the idea that someone might want to install a port which has a hundred dependencies. Under your new scheme, MacPorts would go and fetch 100 distfiles, and then the user encounters an error while building the 3rd dependency and gives up, having unnecessarily waited for the download of 97 other distfiles he'll never get to. Of course we don't want users to encounter build errors, but sometimes they do.
> 
> The real question is: why does it matter to you whether MacPorts fetches right before building each port, as it does now, or all up front as you suggest? It'll take the same time either way (except the added time in your version to compute what to fetch). The useful idea suggested in the ticket is to fetch in a separate thread, while MacPorts is building other already-fetched ports, and this would indeed reduce the time it takes to build, but that would probably be difficult to implement without rewriting a lot of how MacPorts base works since we don't currently use threads. And even if I felt comfortable doing so, which I don't because I'm not that familiar with MacPorts base code or C or threads programming, I would be wary to do so, since it more or less works now, and I don't want to break it.
> 
> However, it should be easy to build a shell alias or function ("port rfetch"?) to do whatever prefetching you want, which you would invoke as desired.
> 


I guess there *are* good reasons it cannot be the default then :)

As for why it matters, take the following: I want to install something that involves gcc. I type "port install thing". Thing invokes gcc (no I did not check beforehand that gcc was a dependency). gcc takes 2 minutes to download but several hours to compile. In 2 minutes I am sure I still have an internet connection. In several hours, I may be on the bus or wherever but the port install process is still ticking away in the background and will fail when it gets to the next thing after gcc.

So it matters only when the build time is >> the fetch time. Does it *really* matter? No. Just a nice-to-have. I am less worried about a 45 second delay due to pre-fetching if it will take 3 hours to compile. But as you say, it may not be worth it if it slows down the build of "easy" ports.

Daniel