Been in pretty much that same position. First, it's not your job to wait for crew coming in late. They need to catch up. What we've done in the past is sail to Maho and spend the night there. The next day start a bit slow and take your time. Go to West End. Your friends can take the ferry when they get in and meet you there. Depends a little on what time they get there but a short sail over to Norman for the night. Everyone will be happy. Did this a couple of years ago and it worked out just fine.

Make sure you tell C&I that you have two crew coming on the ferry and that they will be coming with you on the boat back out (i.e., not leaving Tortola on the ferry). That's a little unusual so make sure they understand your plans.