Ok, well, the 180 thing seems logical (because I'm flipping something by 180 I guess the others have to change as well). Maybe
So, my fix is to use 'Set Rotation' to quickly rotate the gameobject after each 90 degree rotation. Since this is not the visual, it's parented to the visual object, I can quickly unparent, set rotation, reparent.
If your object is visual though (like a dice) then you couldn't do this
Hope that helps someone.