I think I worry about it dying if there doesn't maintain a steady stream of posters is all. Huge is really not my style either, that's too much for me to handle really. Last time around we only had about 2600 members and most of them did not post so it wasn't insane.
I'm with you, quality over quantity. Just not so small that people abandon it for other sites and it becomes a deadzone - that is the only part I worry of =) I was mostly worried the other day because sign ups and post counts were getting smaller everyday, but I'm a worry-wart, that only lasted a couple of days, things seem to be pretty steady now again these past few days.
In the end I think you are right, those that want a circus go elsewhere and those that want a more adult community with quality posts seem to be hanging out here - so I think that's cool and definitely better then a circus.
So naa, I don't want a huge population, just a population steady enough not to become a dead board