It actually I think it would be doable on a big "if". It is about a 4 minute walk from Le Petit to Rue Happy Bay (Cool Heure Restaurant right there, gets great lunch recommendations). Walk up Rue Happy Bay to the gate, if open keep going straight. If not, go to the right and then left where the wall starts heading towards the beach. Walk up along the wall and not sure if still there but there used to be a spot in the wall that was easy to climb over It was in the first 30 or so feet. From there the walk is easier than from Friars. As for legally walking up that way, hard for me to imagine that anybody would care since almost everything in that area is abandoned.

If you have to walk all the way around to Friars then I would not consider that, but if you are able to access via Rue Happy Bay it is probably easier than the goat trail walk from Friars.