Ok, solved most of it, the baking had put small elevated walkable areas on objects that were supposed to be not walkable, making the results confusing.
But helped me do a lot of cleanup as I found if you use Nav Mesh Calculate Path action before you make the agent move, you get your source pos from your character, and target pos from the mouse click. Then there's a path found and path not found result in that action so you can get rid of all the out of bounds or non navmesh clicks easily as it won't find a path if it's not a click in the walkable navmesh.
