I own the 28LS, 35BR, older 45, 40-80BR, 75-150BR and have shot extensively with the 55 and 80. Here's my take:
From a versatility standpoint and being able to frame your subject exactly how you want, the zooms are very convenient. In most shooting situations, if properly focused (and properly trimmed if you use AF), you will find little difference at f/5.6 or higher f-number on the wide end of the 40-80 zoom and f/8 or higher f-number on the long end. Wide open, the zoom is not as good as the primes at the same aperture as the zoom which is to be expected since the primes are being shot stopped down while the zoom is wide open - no surprises there. Where there is a difference is when shooting into a bright light source where the zoom has more flare and it is an aperture shaped flare which shows up with bright point light sources. On the wide end, the 35BR is better than the 40-80 in all shooting conditions, the 28 is worse than the zoom in all shooting conditions but then again that is a completely different lens and dramatically wider so it isn't really a valid comparison. If I were to build a prime kit in your range of interest I would probably go with the 35, 55 and 80 as that gives you a real nice range (the 45 is just not that wide although the new one is excellent)
I totally agree with Paul on the size and weight of the zooms - they are big and heavy.
As an aside, the 75-150 is absolutely awesome in it's entire range.
Hope this helps.