Island Yachts is based in Red Hook so the beat up the South side of STT from Charlotte Amalie is not an issue.
I have sailed out of the harbor where CYOA is based many times and there are really 2 options - long tacks in relatively comfortable seas or short tacking closer to shore in somewhat protected water half the time and short choppy swells the other half - the ferries power through the short choppy water along the inshore route most of the way and that is the roughest option most days - many charter boats motor sail with the main somewhat on that same route just to get it over with on the shortest route, IMHO doing it under sail is the most comfortable choice but certainly less direct - unless you get a wonderful Southerly on a summer day.
Pillsbury Sound can be a rolly ride but it is usually over on one starboard tack for 30 min. into the protected waters behind STJ. Once you are abeam of Two Brothers you are out of the prevailing swell.
I sailed upwind out of Red Hook daily for almost 25 years and it was rarely anything beyond a pleasant exhilarating morning sail. The passage through the Durloe Cays can be challenging under sail if the current is running against you.