We did a day trip to Anegada from VG early January and while there was no sargassum, the swells were incredible. The ferry over was borderline scary and the water was so rough near Ann's and Cow Wreck we didn't dare step foot in the water. It was unpleasant to even sit on the beach due to the wind whipping the sand. We drove the whole island and really didn't find anywhere I felt safe swimming. We regretted spending the time and money to get there since there wasn't a whole lot to do with the wind what it was. Last year seemed particularly rough but who knows what this year will bring either.
It sounds like you are looking for more relaxing time...in which case, I would consider just staying on VG. More often than not, we overplan and overbook, only to look back and say "we should have just relaxed more".
Take the ferry from VG to Jost on Wednesday instead. A few painkillers solves everything!