I spent a(nother) few hours last week studying lgroup partition loads - the not so famous lpl's - and after an email to the OpenSolaris Performance Community I got a couple of very good replies. So I decided to write it up here and add some documentation to this. You can check out the original thread here.
(a) lpl's are the intersection of lgroups and CPU partitions. When you have CPU partitions cutting across lgroups, you need to constrain the set of CPUs where a thread can run but also consider the lgrp's CPUs that are within that constrained set.
(b) lpl's are used to determine the load average of lgroups, which is used by the dispatcher to load balance threads around.
(c) the number of lpls an lgrp can have associated is limited only by the number of logical CPUs it contains. Leaf lgroups will have associated leaf lpls. Just as the lpl is the result of intersecting an lgroup with a CPU partition, the lpl hierarchy is the result of intersecting a cpu partition with the lgrp hierarchy.
(d) lpl topology is created by lgroup code and triggered by processor set instantiation / destruction (and by creation of the default CPU partition)
lgroups and processor sets are key concepts when scheduling and dispatching threads, but they can very easily be orthogonal to each other. Processor sets cutting across lgroups could make the entire MPO effort go away by making threads run in completely different nodes. This would add latency to memory access operations and bring performance down. lpl's make sure that doesn't happen. "Possible orthogonality" is a good way to describe it, IMO, since a partition may or may not span across lgroups.
