Hi, i tested a little and i think this is ok :
i made a sprite from the ladder and placed this above the ladder with a box collider 2d (disabled) and turned off the sprite renderer.
Then on the ladder fsm in the on ladder state i use the custom action Enable Collider 2D to enable the collider and in the wait(Ofll Ladder) state i disable the collider.
And also on the jump fsm on the do jump state i have a Enable Collider 2D to disable the collider.
If you have several ladders drop all the colliders as a child and use Incl Children.