Both the Azure Art Hotel and Horny Toad Guest House are a few blocks' walk to the main road that goes through Simpson Bay, where there are a plethora of restaurants to choose from. Both are RIGHT on the water and are smaller properties. I stay at Horny Toad a lot and they often have early check-in available. I'm sure souvenirs are available in Simpson Bay although I can't think of a place off the top of my head.

LINK to Azure

LINK to Horny Toad