I swim at Lindberg usually 4 or 5 days a week all year -- there are not a lot of days that I see the anchored boats rolling - if the wind is due South it can have a swell though, which you will already know from the SVI sail back. During the busier months there are often 4-6 boats there overnight - the SE cove is the calmest spot but does get backwinded from the hill. There was a sloop anchored there for almost 3 months this winter and I spoke with the young couple often, gave them a ride to town on occasion and they felt it was very calm most days and good holding.
Brewers is certainly a calmer, flatter anchorage, I haven't heard of any restrictions except for boats that look to stay there permanently and try mooring. The food/drink trucks there do have daytime service - it is not very convenient to the airport for a pickup except by taxi or rental car. It probably would only add 20 min. or so over Lindberg for a morning motor back to Frenchtown.