[MacPorts] #71711: portindex: allow thread count override via cmdline param and new macports.conf property
MacPorts
noreply at macports.org
Wed Jan 1 22:06:40 UTC 2025
#71711: portindex: allow thread count override via cmdline param and new
macports.conf property
-------------------------+-----------------------
Reporter: mascguy | Owner: (none)
Type: enhancement | Status: assigned
Priority: Normal | Milestone:
Component: base | Version: 2.10.5
Keywords: | Port: portindex
-------------------------+-----------------------
On older systems, the process of thread creation [within `portindex`] is
rather slow. However, older systems typically have no more than 8 CPU
cores. And often they have even less, such as 4 or 2. But with low cores
counts like this, thread creation overhead is generally fine.
This is particularly true when scanning the entire ports tree, where the
scanning process time makes that negligible.
But when used for a directory tree containing a minimal set of ports -
such as what one might use to stage port fixes/updates - that overhead
suddenly takes the vast majority of the time.
And on a system like mine - a MacPro2012 with 24 logical cores - that
overhead suddenly becomes excruciating.
A great test case involves creation of a new, empty directory, and then
running `portindex .` from in it. Since there are no ports to process, the
execution time will be almost all related to job setup.
With some extra debug logging, that test scenario results in the
following. (Date stamp format used: `%Y%m%d-%H%M%%S`.)
{{{
$ pwd
/var/tmp/portindex-test
$ ls -a
. ..
$ gtime portindex .
portindex: startup: 20250101-170008: 1735768808609ms
portindex: packages loaded
portindex: mportinit done
portindex: script starting
portindex: script directory stuff 1
portindex: script directory stuff 2
Creating port index in /private/var/tmp/portindex-test
portindex: set fd
portindex: starting threads: 20250101-170009: 1735768809012ms
portindex: init_threads: start
portindex: worker_init_script: start: 20250101-170009: 1735768809036ms
portindex: worker_init_script: start: 20250101-170009: 1735768809344ms
portindex: worker_init_script: start: 20250101-170009: 1735768809659ms
portindex: worker_init_script: start: 20250101-170010: 1735768810044ms
portindex: worker_init_script: start: 20250101-170010: 1735768810360ms
portindex: worker_init_script: start: 20250101-170010: 1735768810672ms
portindex: worker_init_script: start: 20250101-170010: 1735768810983ms
portindex: worker_init_script: start: 20250101-170011: 1735768811293ms
portindex: worker_init_script: start: 20250101-170011: 1735768811607ms
portindex: worker_init_script: start: 20250101-170011: 1735768811929ms
portindex: worker_init_script: start: 20250101-170012: 1735768812336ms
portindex: worker_init_script: start: 20250101-170012: 1735768812642ms
portindex: worker_init_script: start: 20250101-170012: 1735768812964ms
portindex: worker_init_script: start: 20250101-170013: 1735768813371ms
portindex: worker_init_script: start: 20250101-170013: 1735768813684ms
portindex: worker_init_script: start: 20250101-170013: 1735768814021ms
portindex: worker_init_script: start: 20250101-170014: 1735768814437ms
portindex: worker_init_script: start: 20250101-170014: 1735768814807ms
portindex: worker_init_script: start: 20250101-170015: 1735768815117ms
portindex: worker_init_script: start: 20250101-170015: 1735768815420ms
portindex: worker_init_script: start: 20250101-170015: 1735768815780ms
portindex: worker_init_script: start: 20250101-170016: 1735768816191ms
portindex: worker_init_script: start: 20250101-170016: 1735768816609ms
portindex: worker_init_script: start: 20250101-170017: 1735768817022ms
portindex: init_threads: end
portindex: final work
Total number of ports parsed: 0
Ports successfully parsed: 0
Ports failed: 0
Up-to-date ports skipped: 0
portindex: done: 20250101-170017: 1735768817305ms
3.93user 4.86system 0:08.75elapsed 100%CPU (0avgtext+0avgdata
143188maxresident)k
0inputs+0outputs (0major+36046minor)pagefaults 0swaps
}}}
The run takes nearly 9 seconds, with no work done processing ports.
So... it would be highly desirable to be able to override the default
thread count (which appears to use `buildmakejobs`), via a new config
property. A cmdline parameter might be nice too, but either option would
be great!
--
Ticket URL: <https://trac.macports.org/ticket/71711>
MacPorts <https://www.macports.org/>
Ports system for macOS
More information about the macports-tickets
mailing list