Hi,
I think, it's because you have two different mode, one is accessing the rotation of the object directly, and the other is incrementaly rotating, so when you switch, you can't keep the rotation as it is. Or you are on one hand changing the look at target, and on the other hand actually rotating the object, which also leads to inconsistency.
the smooth look at should always be active, and then you should only change the target.
Bye,
Jean