No, you cannot hike to the conch mound. Getting there by land would require crossing Budrock Pond. It is shallow, but the bottom of it is extremely soft - as close to quicksand as I have ever seen. I have tried going that way.
Hiking the very south side is also impossible - it is solid mangrove thicket
.
