If someone in the group has some small power boat operating experience and the conditions are good and will remain so during your trip, my opinion is you will be fine in something around 20'. Of course, I assume you are kidding about all heck breaking loose and attempting to swim to shore.

Life involves risks, take some prudent ones (NOT with the BVI ferries)!