Well that is actually VERY strange. It worked as you mentioned exactly, problem is, I had the same exact setup on the ball with the tag being "Bricks" and it would destroy the object in question.
However, that didn't work, it worked when the brick had the code. If someone can explain the difference it would help greatly.
That said, there is a new problem, for some reason the ball bounces of some of the bricks and some no. Not sure why, I am using the same prefab.
I did the setup with a wait "0.5" before the destruction of the brick so the ball could bounce.
Thank you very much Zeldag!