You have to reference the object that's in the scene. The Project Hierarchy and Prefabs are like a library, when you spawn an object it goes into the library, copies the book and throws that copy in the Scene. You can't destroy the books in the library, only the copies..
Since you can only destroy the copies you have to design the original prefab to cleanup itself at some point or have the a manager of some sort clean up the copies, like a Pooling System.