Go to Charlotte Amalie "Tortola Wharf" first to check whether there is a ferry leaving from there. Luggage might take some time to come out and then the taxis have you wait until they have the van filled to the max before the driver will take off. Do not believe the taxi driver that there is no ferry from Charlotte Amalie. The driver did that to me and there were 2 boats leaving and he only stopped there for me after a long argument. With the traffic to Red Hook you might not make the ferry. I would have missed the ferry from Red Hook.