Not crazy at all. I've done this three times. When five ships are in port, St. Thomas is a nightmare. The taxis are gridlocked, the beaches are shoulder-to-shoulder, and downtown is a sweaty mess. Staying on the ship is the best "secret" in cruising. The pool is empty. The bartender remembers your name. Do it.