We usually spend 2 days on-island before climbing on the charter boat. I'm usually ragged out from getting stuff wrapped up at work and need a little decompression/rest time before hitting a charter.
Also, we usually spend a nite on-island before flying out.
There seems to be too much to do to make an early flight out that day, and we don't want to be getting to the destination airport after midnight and exhausted.

After all, this is supposed to be a vacation!