Hi,
you should never parent a non rigidbody to a rigidbody, alwasy use joints. If you need a hierarchy, set your childs as rigidbody but set them to kinematic and add a fixed joint.
the other solution would be to force position and rotation of a gameobject to a Rigidbody on late update, so the parenting is soft and then mouse will work.
Let me know if you still struggle with this.
Bye,
Jean