In the Reset() function (where you set the default values for your variables), that you override from the inherited FsmStateAction, you have to set 'UseVariable' to true like that:
public override void Reset()
{
min = new FsmVector3() { UseVariable = true };
max = new FsmVector3() { UseVariable = true };
storeResult = new FsmVector3();
}
(it would probably also be possible to write min.UseVariable = true; later in the script, but I couldn't think of any use-case for that).
Your approach doesn't work, since you create a new FsmVector3 with the name "None", which has nothing to do with the 'None' state of FsmVariables.
You can also find more examples or infos on creating custom actions
here, where I also provide an ExampleOverview.cs that shows in more detail how this can be achieved.