Possibly you can double up your rigidbody controller to have a regular character controller too? Don't know if that's a bad thing, but basically never do any input to move around other than to move you up on the Y axis? Or maybe the rigidbody woudl refuse to budge to a character controller command. If you can get this kind of combo to work, you are then able to spoof gravity / make a return back to the ground by just moving the controller back down until you are grounded.
Alternatively: You said gravity is the issue with your rigidbody? So maybe just turn it off? There is an action which is something like 'Use Gravity' or similar (just search gravity in the action list) and toggle it on and off like that.