Of course - if you've only been used to 4-5m2 kites with a lot of grunt (camber and to some extent low AR), you will adapt and ride with your kite higher, or on shorter lines - which can compensate for this, IMO, bad behaviour.
But otherwise, it is very evident that they will pull you "out of line" when flown deep, when you go vertical on the waveface (up or down)
I have never seen a 4-5m2 kite that does not "appear" to drift well, as in these winds it is not the kite that drifts well.
Simply because you can not (extremely difficult) outrun the wind speed, so you never get slack lines except in lulls, thus no true "drift", so all really small kites seem at first glance to drift really well usually
Even though they might not - but you will never find out, as not used in light winds where you outrun the wind often....
So if used to these sizes or adapting a different style and/or lines, one can get away with it, true
But for me it is a PITA
that you have to change style so violently when you go from say a 7 to a 5m2 with normal linelengths (or from a 6 to a 4m2), and when you change between these two during a day (which we do really often here, as wind is around these "sizes" a lot), it gets even worse.
Some, riding with clean side or sideoffshore winds, keeping the kite parked or high, or with really short lines, wont notice.
But everybody else got the same problem with too grunty 4-5m2 kites, which I find most kites are...
Most have never tried a kite with huge depower (lower camber) in these sizes, so they dont know what they are missing
That could be another reason why there are so many different opinions on this, besides personal taste of course.
On twintips it is just opposite - we mostly want good grunt in these smallest kites, as the kite will then "behave" easier, and not be as on/off and erratic as a more low cambered kite might be
PS: It is NOT a question about having a kite that is too big....
As the kites we use are perfectly powered when going straight and upwind, not too much nor too little - as we choose the size after this, solely.
But when you ride up and down the wave, and you fly the kite all the way down low in the powerzone - they pull way too much (and gets even worse with lower AR although easier to fly) and will hinder any free active waveriding
Whereas medium sized kites around 7m2 (for the average rider) dont have this tendency at all, because of the lower wind speed, and fits mostly bullseye for many riders