"He was well outside the bay. He was out in open water. But that's an interesting question - if he was out swimming in the Drake channel would that be my fault 'cause I should have seen him? Where is the limit - if it exists - of where the boater is responsible for the swimmer?"

I don't think it is about being responsible for the swimmer - more a matter of being responsible for the operation of the vessel.

Certainly a swimmer in open water should do their best to be aware of the possibility of boat traffic but I think the responsibility in the end falls to the vessel.

http://navruleshandbook.com/Rule5.html

Keep a proper lookout.