Hi.
I have not worked with vr yet but i do use vr.
1st thing comes in mind is to have a 'empty object' to follow controller(real Hand)
Then have the VR hand object 'follow' the 'empty object' (real hand)
The follow part can be more tricky to do
the vr hand should have colliders and probably you will need some raycasts to for example stop following toward, but possibly still follow sideways.
But again, i have not tested vr in unity yet