It's a weird problem to have. Are the invisible walls totally needed?
Anyway, a quick fix would be to change your player's collision layer when he's airborne.
If he's in the air, assign a collision layer that ignores invisible walls. When he lands on the floor, set his layer to something that obeys the walls.