In general, it's recommended to fly over because even if you buy a ticket (at https://www.puertoricoferry.com/en/) you aren't guaranteed a seat. Puerto Rico residents are given priority, so it's entirely possible that you will be bumped and cooling your heels at the ferry terminal with 3 kids. I will also say that it seems that the ferries are a rough ride. Even if you don't get seasick, there's a good possibility that someone else on your ferry will! They're not really like the USVI-BVI ferries--I've seen video and on the passenger/cargo ferries you're stuck below decks with no windows and when it starts rolling...

You can fly from Ceiba, Isla Grande, and SJU on Cape Air (SJU only), Air Flamenco, and Vieques Air Link. SJU flights are usually a little more expensive but the way the flights work, you might as well just go for the times that are best for you because they all cost about the same depending on origin. Once when we went to Vieques with a group of 5 adults I made a whole spreadsheet of times and costs and we flew out of Ceiba, but I am admittedly a little nuts. On our most recent and our upcoming Culebra trip we're doing Isla Grande on the way down and Cape Air to SJU on the way back. Since you have two lap-sized kids, I'd check with the airlines about the best way to do it. They are genuinely tiny planes and the Culebra landing is an adventure!

At the end of the day, what you get in exchange for all that planning and math is a really spectacular vacation. Culebra is right at the edge of not developed enough and developed just enough. It's certainly not overly busy! Because so many people from Puerto Rico head over with their families for their vacations/long weekends, there are probably lots of places to stay that will be very kid-friendly.