Yes, it is a bit rough out there for a dinghy. I usually anchor in 10-12 feet inside Anse Marcel and dinghy in to the Capitainerie. I've found it to be rolly in Anse Marcel so only stay long enough to clear in/out and then go to Grand Case.
Another option is to anchor in Grand Case and take a bus to Marigot ($2) then clear in at Island Waterworld on the French side for a couple of Euro.