Most people take the ferry from Charlotte Amalie, which is closer to the airport. The ferry schedule is limited, so you may have to wait a couple hours. Most people take the ferry to Road Town, which is about 1/2 hour by cab to Trellis Bay; plan on one hour from West End, if that is where your ferry goes. Returning you clear Customs twice: at the port and the airport. You need to be at the airport at least 3 hours before your flight. You will lose a day on your return, unless you fly or take a water taxi.
If you fly to Beef Island, C&I is generally shorter, you walk five minutes to Trellis and wait for the hourly ferry. This is, by far, the better way to get to Scrub Island.