As long as you have a battery between the panel and the charger, there will not be any problems.

The problem is when the solar output drops some, but not fully, then the charger can lock up until the panel voltage goes very low.