You can't go wrong with Little Harbor OR Diamond Cay...BOTH are great anchorages in my opinion and much better anchorage/mooring options on JVD for quiet/peaceful.

In your case I would just choose based on what you and your group want to do off the boat...bubbly pool hike, Foxy's taboo and B-line, then Diamond Cay is the play - quiet boat time with dinner options and slightly cheaper (maybe) cab ride to White Bay, then Little Harbor is the play.