Okay, good to know. I've been warned about raycasts being heavy in the past, so I generally try to find workarounds that are as easy going as possible.
What I did, which isn't too much of a change to your system, was instead of using raycasts, use a large (32m in diameter) trigger volume, which I call the "radar" volume, that follows the enemy around. Anytime the player enters the volume, the enemy begins keeping track of the distance to the player (but remains in the idle/non-combat state for the time). If the player gets >Xm from the enemy, the enemy switches to an "assessment" state, and decides to do from there (some run away, or some will chase the player and attack when at another specified range).
I have a second volume within the radar volume which is my "sight" volume. If the player enters the sight volume, it acts like your raycasting setup, and the enemy immediately enters it's "assessment" state regardless of player distance.
Regarding the breadcrumb system, what I was going to do was basically have a single breadcrumb as a child of each enemy. When the enemy is tracking the player within the sight volume, the breadcrumb object follows the player around. Once the player moves out of range (or sight, but even I may need raycasts for that!) the breadcrumb object remains where the enemy last saw the player, and the enemy enters a "search" state. The enemy will proceed to the breadcrumb, and then look around for the player. The rest is more or less self-explanatory.
If you do use that approach, and you have to use raycasts for line of sight detection anyway, maybe you can use the raycast as a secondary system, instead of always being active. That way your AI is only casting when in the players immediate vicinity?