You’ll have that same experience where there are currents and the winds die down or in some cases when backwinded.
Tightening the pennant doesn’t help because the mooring balls have a certain amount of scope to make up for the tides and seas so your hull will still get the line caught along the keel.