Others here will offer their opinion, but it is not just the swell height that you should be concerned with. The swell direction, and the period are important too. The longer the period, the more energy the swell carries.
Depending on your crew and vessel (monohulls are more susceptible), a 4 ft swell at 14+ second period can be uncomfortable for many.....especially in Cane Garden Bay. Other places that are dangerous in north swells are the Baths and Cooper Island. Landing a dingy in White Bay (Jost) can be dangerous too because of the swell reflecting off of Tortola.
The veterans here, with much more experience than I, will add their valuable two-cents.
.
