I would definitely count on NOT making the 4:30 Speedy's, unfortunately. You will most likely be getting off the boat with locals who've got shopping, etc. with them -- it will take you a bit either way to clear.

Not sure if you want to spend the $$, but you can fly one-way from STT to VG on Air Sunshine for $150 each. Daily flights at 3:30 and 5:30 (probably the most realistic).
airsunshine.com