Sadly, Captain Oliver's in still in shambles.
In Oyster Pond there are 2 very good restaurants - BZH and Oasis. Both are at the bottom of the hill just after you cross over to the French side. As far as a baked goods, Bread & Butter is right next to Oasis and has a bakery and a very nice selection of other products. Plus excellent coffee! Several nights a week they also have freshly roasted whole chickens, but you must speak for one in advance.
And there is always the Big Fish just before the entrance to Oyster Bay Beach Resort.
Hopefully these options will save you a long drive for at least a few of your nights!