You can take the ferry from Road Town to Anegada on Friday early morning (6:45am) or Sunday (8:00am) -- I think there may be options for flying there but it's not something I've ever done so I can't weigh in on that. You can return at 5pm both days to Tortola.

There may be other options besides the ferry for JVD -- but you can ferry there from the West End pretty easily. Where are you staying at CGB?

Have fun...