Definitely try Wonky Dog for a gourmet experience on Anegada.
Lobster Trap is now owned and operated by Anegada Beach Club so it has a slightly more gourmet menu than Potter's and much of the food is the same as the Beach Club.
If you are choosing between Lobster Trap and Potter's, I would recommend Potter's. Sam is the chef and his traditional Anegada menu is very good and his personality and post-dinner limbo action makes it a must-do for us!