Agree that Savannah is probably your best bet for swimmable water during swells. Plenty of reef break there. It won't be dead calm but it will be manageable.
There's always pool at Gunnies! And you never know what goodies they are cooking up in The Bond. You couldn't spend a more authentic afternoon on VG.