Hello everyone ! I keep lurking this forums for help and usually I end up succeeding with whatever I needed through previous posts, having that said I just can't seem to fix this issue :
The project was more complex but I tried recreating an extra simple version of my character to try and pinpoint the problem but to no avail. As soon as the character land anywhere else (besides the starting point) it starts glitching as seen on the gif (sometimes later stabilizing)
I am basically Raycasting towards the floor, storing the Hit Normal, created a vector3 thats 0,1,0 to get vectorUp, Get Quaternion From Rotation (from vectorUp to hitNormal) storing the result and setting the rotation of my child (the character itself) with this result.

I am also storing if the Raycast hit anything or not, if it didn't, it'll change the State to a new one where it resets the character's rotation so it's standing vertically until the Raycast gets a hit once again. (these 2 are always jumping from one to another as soon as the glitching starts, logically)

The Empty Object "pivotAngle" is centered to the character while the "lePivot" is standing in front of it on the Z axis, this is used in combination with Look At 2 so lePivot is always oriented towards where the character should turn with Get Axis Vector while the character has a Smooth Look At towards lePivot so then it moves locally in the Z direction.
Thanks in advance, if any further information is needed just say so.
