So I've tried multiple ways now, the latest version i am using a big invisible button, with global events being triggered. The lag i am referring to, turns out, is the button firing the event on mouse up!
So instead of Ui click event, i used pointer down event.
And finally, it's working, Lag free.
Thanks so much guys.