[MacPorts] #71711: portindex: allow thread count override via cmdline param and new macports.conf property

MacPorts noreply at macports.org
Wed Jan 1 22:06:40 UTC 2025


#71711: portindex: allow thread count override via cmdline param and new
macports.conf property
-------------------------+-----------------------
 Reporter:  mascguy      |      Owner:  (none)
     Type:  enhancement  |     Status:  assigned
 Priority:  Normal       |  Milestone:
Component:  base         |    Version:  2.10.5
 Keywords:               |       Port:  portindex
-------------------------+-----------------------
 On older systems, the process of thread creation [within `portindex`] is
 rather slow. However, older systems typically have no more than 8 CPU
 cores. And often they have even less, such as 4 or 2. But with low cores
 counts like this, thread creation overhead is generally fine.

 This is particularly true when scanning the entire ports tree, where the
 scanning process time makes that negligible.

 But when used for a directory tree containing a minimal set of ports -
 such as what one might use to stage port fixes/updates - that overhead
 suddenly takes the vast majority of the time.

 And on a system like mine - a MacPro2012 with 24 logical cores - that
 overhead suddenly becomes excruciating.

 A great test case involves creation of a new, empty directory, and then
 running `portindex .` from in it. Since there are no ports to process, the
 execution time will be almost all related to job setup.

 With some extra debug logging, that test scenario results in the
 following. (Date stamp format used: `%Y%m%d-%H%M%%S`.)

 {{{
 $ pwd
 /var/tmp/portindex-test

 $ ls -a
 .       ..

 $ gtime portindex .
 portindex: startup: 20250101-170008: 1735768808609ms
 portindex: packages loaded
 portindex: mportinit done
 portindex: script starting
 portindex: script directory stuff 1
 portindex: script directory stuff 2
 Creating port index in /private/var/tmp/portindex-test
 portindex: set fd
 portindex: starting threads: 20250101-170009: 1735768809012ms
 portindex: init_threads: start
 portindex: worker_init_script: start: 20250101-170009: 1735768809036ms
 portindex: worker_init_script: start: 20250101-170009: 1735768809344ms
 portindex: worker_init_script: start: 20250101-170009: 1735768809659ms
 portindex: worker_init_script: start: 20250101-170010: 1735768810044ms
 portindex: worker_init_script: start: 20250101-170010: 1735768810360ms
 portindex: worker_init_script: start: 20250101-170010: 1735768810672ms
 portindex: worker_init_script: start: 20250101-170010: 1735768810983ms
 portindex: worker_init_script: start: 20250101-170011: 1735768811293ms
 portindex: worker_init_script: start: 20250101-170011: 1735768811607ms
 portindex: worker_init_script: start: 20250101-170011: 1735768811929ms
 portindex: worker_init_script: start: 20250101-170012: 1735768812336ms
 portindex: worker_init_script: start: 20250101-170012: 1735768812642ms
 portindex: worker_init_script: start: 20250101-170012: 1735768812964ms
 portindex: worker_init_script: start: 20250101-170013: 1735768813371ms
 portindex: worker_init_script: start: 20250101-170013: 1735768813684ms
 portindex: worker_init_script: start: 20250101-170013: 1735768814021ms
 portindex: worker_init_script: start: 20250101-170014: 1735768814437ms
 portindex: worker_init_script: start: 20250101-170014: 1735768814807ms
 portindex: worker_init_script: start: 20250101-170015: 1735768815117ms
 portindex: worker_init_script: start: 20250101-170015: 1735768815420ms
 portindex: worker_init_script: start: 20250101-170015: 1735768815780ms
 portindex: worker_init_script: start: 20250101-170016: 1735768816191ms
 portindex: worker_init_script: start: 20250101-170016: 1735768816609ms
 portindex: worker_init_script: start: 20250101-170017: 1735768817022ms
 portindex: init_threads: end
 portindex: final work

 Total number of ports parsed:   0
 Ports successfully parsed:      0
 Ports failed:                   0
 Up-to-date ports skipped:       0

 portindex: done: 20250101-170017: 1735768817305ms

 3.93user 4.86system 0:08.75elapsed 100%CPU (0avgtext+0avgdata
 143188maxresident)k
 0inputs+0outputs (0major+36046minor)pagefaults 0swaps
 }}}

 The run takes nearly 9 seconds, with no work done processing ports.

 So... it would be highly desirable to be able to override the default
 thread count (which appears to use `buildmakejobs`), via a new config
 property. A cmdline parameter might be nice too, but either option would
 be great!

-- 
Ticket URL: <https://trac.macports.org/ticket/71711>
MacPorts <https://www.macports.org/>
Ports system for macOS


More information about the macports-tickets mailing list