[Aboriginal] Load limiter patch

[Rob Landley]

On 10/01/2013 06:16:06 AM, Patrick Lauer wrote:
> Hi all,
> 
> I noticed some, ahem, suboptimal behaviour on machines with lots of
> memory when FORK=1 is used.
> 
> Problem: the current algorithm sets the number of parallel jobs to
> memsize/512MB. On a machine with e.g. 32GB that's 64 jobs, since there
> are 17 targets currently that gives a per-target parallelism of 3  
> which
> adds up to 51 parallel jobs ... on a machine with 4 or 8 CPU-cores.
> 
> You can imagine how that's, err, not happy.

Ok, now that I've read your actual patch, let me write a different  
response. :)

When you said "load limiter" I thought you were using the "-l LOAD"  
option of make, I.E. --load-average. But you're just capping CPUs, and  
the way you're doing it is an integer division:

   CPUS=$(( $REAL_CPUS / $TARGET_COUNT ))

Means that if REAL_CPUS is 8 and TARGET_COUNT is 12, you get 0 (so each  
target basically gets $REAL_CPUS). Or if REAL_CPUS is 16 and  
TARGET_COUNT is 12 you get 1 and leave 4 processors idle...

That's why I didn't do that; integer rounding gets ugly no matter how  
you look at it.

It's possible that telling make to do its own load balancing will work  
out better. I honestly don't know, see "don't trust anything the FSF  
implements to not be broken", but that looks like the thing to  
investigate?

I also note that what I tend to want is to max out CPUS for the  
host-tools and simple-cross-compiler i686 builds, and then CPUS=1 for  
all the other target builds so I get a coherent log out of them.  
(Building with CPUS=1 means the log of the build output is harder to  
read if you care about that sort of thing.) That still gives pretty  
reasonable parallelism on the systems I'm building on, but shouldn't be  
the default because if you have a 64 processor monster it won't take  
advantage of it at all. I've been meaning to do a knob for that, but  
haven't yet. I generally just "CPUS=1 FORK=1 more/buildall.sh" to do  
release builds...

Sigh. And these days, FORK=1 should probably be the default and I  
should change the variable to NOFORK=1 to switch it off...

Rob
 1381889267.0