SE winds in either place should not be a issue. The reason some warn against overnights in White Bay JVD is squalls or thunderstorms. You kind of have to ride through one there to understand how quickly things can so bad. When it does go bad boats can quickly end up damaged on the beach or dropped hard on their bottoms in the wave troughs. I only stay there overnight in perfect weather with no chance of evening storms. That means I rarely stay there.
G