We tried it...once. the private property holders on the island do not welcome hikers. There is this one little guy at the top of the hill that will pop out of his hut and defend his turf.
It was not this way before the island resort got big.

Hop in your dink on a calm day and head over to Salt.