I don't think you have to stay in your resort in St Lucia. We left Anse Chastenet and had dinner in town etc. But!
The driving there is onerous. Roads are long and dark and not knowing your way around is a problem. And they are also long in the daytime! Long and curvy and mountainous.
Good luck
cheers
irina