A few thoughts: Are you thinking of hauling your kayak from Setting Point to the SE part of the island and launching it there? Seems like a lot of trouble... If anything I could see doing westbound downwind one way back to setting point.
But fundamentally I don't see the appeal of Conch Island. Reminds me of the photo of a giant pile of buffalo skulls from the 1800s.