We did it once. Started from St John to Christiansted about 6-6.5 hours in a small Island Packet, a reach out and back. The town was interesting enough, the fort worth exploring (both are right there at the harbor). The elk horn coral on Buck Island was a little disappointing, it looked dead to us, but certainly not the worst snorkeling we've ever done.

I think a lot of people just aren't into a 6 hour sail and the risk of having to wait for a weather window to get back. Same thing with the SVI's. We much prefer the SVI's but don't regret going to St. Croix - it's a great sail. We didn't make much of an effort to explore the rest of the island, wish we had, maybe next time.