Like noted, yes u could write a CustomActionEditor per enum type, but since Unity and Playmaker already have Attributes for this kind of work, the best solution is using a Attribute.
So instead of creating a CustomActionEditor for every enum mask type, u simply set a UIHintAttribute. Unity already has EditorGUI.MaskField() to handle the actual rendering and i attached the Unity BitMask attribute we use. It would be really nice to also have a "UIHint.BitMask" type in playmaker, to accomplish the same task.
I also suspect this is quite trivial, given that u already have a special UIHint system in place.
Thx Andy