Coming from the USVI, you should stop at Soper's hole. You might be fine going to Spanish town, but it could also be an issue of they were patrolling and saw you and questioned your intentions. Not worth the risk based on stories I've read.

Matt