Hi,
Are you testing this in Editor or is it on the device or published version?
Such feature should not halt your game at all, I guess something very wrong is happening here. Do you have Unity pro? maybe you profile the game and see what exactly is taking time.
else, you need to cut down your problem, disable features one by one to see which one is taking so long. then you can work on that feature to improve it or find another way.
Bye,
Jean