Hi,
Can you try on a simple scene, with a cube looking at another one. Once you master this, you can then apply it in more complicated cases. With this kind of setup, it's usually ( even when you know how to work it out...) something you missed out in the set up that makes it behaving weird, in that case, I'll always start with the most minimal setup I can come up with and build up from there.
If you still have issues, let us know, I can make a small test scene to show it works.
bye,
Jean