Maybe you could use a float clamp on the rotation.
If the float clamp could wrap the rotation that would work.
I tried rotating with the ground normal again using a quaternion, the problem with that is I can only y rotate a navmesh enabled character.
So currently I have my character continually positioned at the navmesh object, trying to y rotate with the navmesh and x/z rotate with the quaternion. The x/z works if I disable to y rotation, and the y with the x/z disabled, but with both it looks rotated backwards. Inverting the quaternion doesn't solve the problem either.
<EDIT>
Wow, it seems to be working now - the rotate with ground normal stuff is on a parent object and the rotate with navmesh y is on the actual model. Thanks so much terri & djaydino for the posts.