Definitely try Wonky Dog for a gourmet experience on Anegada.
Lobster Trap is now owned and operated by Anegada Beach Club so it has a slightly more gourmet menu than Potter's and much of the food is the same as the Beach Club.
If you are choosing between Lobster Trap and Potter's, I would recommend Potter's. Sam is the chef and his traditional Anegada menu is very good and his personality and post-dinner limbo action makes it a must-do for us!
One of our most cherished memories was having dinner there when Verne (original owner) joined us and we heard endless stories about his struggles to homestead there. He lived in a tent for 3 years, and built his own boat to take fish to the St. Thomas market. An amazing story of victory over hardship.
I'm still trying to decide between Cow wreck, Wonky Dog, and Neptunes. My wife and two of our good friends will be stopping by Anegada in about three weeks. I'm leaning towards Cow for the view and good food though Wonky Dog is a close second.