I'm a pretty strong swimmer and I don't think it would be easy to swim back to the boat even if the sails were luffing, especially if you were in a place with a lot of current and wind. The wind would push the boat much differently than it would push the swimmer. And the current would also affect the swimmer and the boat differently. I think you'd want some way to move the boat towards the swimmer. But then again, as the swimmer, you'd probably be highly motivated to swim hard!

I also agree that calling for help on the VHF is probably a good first step if the person in charge is uncomfortable with any part of the process. If someone was unconscious in the water, I'd immediately call for help. By myself, I don't know if I could get an unconscious adult even into the dinghy.