My wife and I mostly cruise with just the two of us and we love it. When we do invite other people, we find that 4 others (6 total) is the perfect amount for us and our boat. Some of our most memorable trips have been with close friends.

As a far as coordinating a larger group, it really isn't a big deal. Most people go with the flow and are happy to do whatever you suggest. People assume that we are the local experts and know the best places to go. Prior to the trip, we always make sure people understand what a typical day is like and that they always have the option of relaxing on the boat with a good book instead of the "group" activity. The only downside are the "required" visits to the famous spots that we have seen many times.