Only creating and destroying will cause stuttering, so either use pooling to recycle bricks (which i recommend) or leave them be if they are standing there doing nothing.
Since the bricks are just empty objects with no variables to reset, pool all the way.