As the transform is by design inherited from the parent, you can either (1) upscale the grabbed objects, or you can keep them out of the hierarchy that scales down, by (2) linking to a bone/bind/mount/placeholder object.
(1) dynamic: right before the scaling happens, unparent the grabbed object briefly, and after scaling parent again. Or static: find out what the scale factor is in runtime, then just set that value to the grabbed object.
(2) you likely have a bone/bind/mount/placeholder object that is used to move a grabbed object to the desired place in the hierarchy. Instead of parenting, move it as child to separate game object, which is linked to the bone/placeholder with location and rotation constraints (it follows the bone/placeholder without being a child).