Not the exact same scene, but the same scripts were applied.
I did a bit of debugging and I discovered that the ray-cast doesn't register as "Not Equal To" until it hits any object other than the object with the script.
So in my original scene which was in a small room, when moving off the selectable item, it would always hit off something (wall etc) since it was full of geometry so it worked fine. When testing in a plain scene that just had a plane and a cube in the air, the problem is apparent. if you look at the cube in the air, then look away from the cube without hitting any other geomery it doesn't register as being "off" the object.
I guess I can solve this by making sure theres always something to hit (in the out door scenes I guess I can just put a huge invisible box collider around the scene).
Any thoughts of a proper workaround? Am I doing something wrong or not using it correctly?!