IndexOutOfRangeException when loading prefab containing PlayMaker FSM

[RiverNyxx]

I'm working in a very large code base that contains a variety of (Unity-provided) async loading for various assets, including prefabs that contain somewhere in them one or more PlayMaker FSMs.

Some of our automated exception reports from our live product show the following truncated stack trace for an IndexOutOfRangeException:

Code: [Select]

System.Collections.Generic.Dictionary`2[TKey,TValue].TryInsert (TKey key, TValue value, System.Collections.Generic.InsertionBehavior behavior) (at <7ba07f088431485bb722f3b3373e87ee>:0)
System.Collections.Generic.Dictionary`2[TKey,TValue].set_Item (TKey key, TValue value) (at <7ba07f088431485bb722f3b3373e87ee>:0)
HutongGames.PlayMaker.ActionData.GetActionTypeHashCode (System.Type actionType) (at <7e9e696ff1a04620968849cd9bbf2ab4>:0)
HutongGames.PlayMaker.ActionData.CreateAction (HutongGames.PlayMaker.ActionData+Context context, System.Int32 actionIndex) (at <7e9e696ff1a04620968849cd9bbf2ab4>:0)
HutongGames.PlayMaker.ActionData.LoadActions (HutongGames.PlayMaker.FsmState state) (at <7e9e696ff1a04620968849cd9bbf2ab4>:0)
HutongGames.PlayMaker.FsmState.LoadActions () (at <7e9e696ff1a04620968849cd9bbf2ab4>:0) HutongGames.PlayMaker.Fsm.InitData () (at <7e9e696ff1a04620968849cd9bbf2ab4>:0)
PlayMakerFSM.OnAfterDeserialize () (at <7e9e696ff1a04620968849cd9bbf2ab4>:0)
UnityEngine.AssetBundle:LoadAsset(String, Type)
...
The only reason I can think of that a Dictionary would throw an IndexOutOfRangeException during an insert is due to a race condition with multiple inserts from different threads around the same time. It appears that this really is possible in PlayMaker, since ActionData.ActionHashCodeLookup (and some others) are static members, and susceptible to race conditions in this very manner.

Consider the above a bug report. Below, a request for how I might deal with/work around this:

Ideally, this would be fixed inside PlayMaker itself. It's the most logical/clean place for it, but since everything is contained inside a .DLL, such changes cannot be made by us.

While waiting for an update that includes such a bug fix, my first thought was to introduce some locking pattern for thread safety inside our loading code, but we don't want to force all loading to use this pattern, as it would effectively turn the async loading into something not far off of sync loading. It's only necessary for loading levels/prefabs/etc containing PlayMaker FSMs.

Any thoughts on how I might avoid this without incurring a significant loss of async loading performance?

[djaydino]

Hi.
does this asset bundle contain reference to new actions?
if so you need to update the main Asset/game as they will not be included in the asset bundle (or playmaker cant find them)

[RiverNyxx]

Hi.
Sorry, I didn't specifically call this out -- as far as I've been able to see, this exception doesn't happen every time. It's actually quite rare to happen. That tells me that it often DOES work just fine, so I believe it's not an issue with an AssetBundle being out of sync with the main asset/game.

Also, correct me if I'm wrong here, but I believe you may be implying here that the exception I'm encountering is a KeyNotFoundException, which would occur when trying to retrieve a KeyValuePair from a Dictionary using a key that doesn't exist in it? However, that's not the exception I'm getting -- I'm getting an IndexOutOfRangeException, and it's occurring when trying to ADD a KeyValuePair to the Dictionary, as the stack trace shows.

[djaydino]

Hi.
I will ping Alex to have a look (he's the creator of Playmaker) .

[RiverNyxx]

Thanks!
Looking forward to hearing about this

[Alex Chouls]

Hi, I have a build that should fix this. Do you have a way to reproduce the error, to check if the build fixes the issue? PM me so we can get you the build to test.

[RiverNyxx]

Good morning,
Unfortunately, I don't have a consistent/reliable way to reproduce this -- all of our attempts to reproduce in a dev environment have failed, but we're getting the exception reported sometimes from our live product.
I fear that my leads will be hesitant to integrate another PlayMaker update so quickly after we'd already done one -- we'd done an update just a few months ago.
While I analyze risk on this, do you have any thoughts about how this might be addressed in our own code with minimal effect to our loading performance?

[Alex Chouls]

What version of PlayMaker did you ship with?

[RiverNyxx]

The current version that is shipped is using PlayMaker 1.9.4.f2
I'm assuming this new build of yours is built on top of the last public release (1.9.5.f3)?
If so, looking at the changelog with my lead, it seems we will be willing to take on the update.

[RiverNyxx]

Hey, just following up on this, as I haven't heard anything in this thread for a while now.

I also sent a PM to Alex about a week ago, but the forums aren't actually showing anything in my "Sent Messages" list, so I sent him another one just this morning (worth noting: It still doesn't show anything in my "Sent Messages" list...)

Just trying to work things out so I can get my hands on this custom build mentioned by Alex.

[djaydino]

Hi.
I will message him

[hianhianhian]

Hey there, I'd like to follow up on this issue too. We're getting similar reports from our live game, and we are also unable to replicate it. Our stack trace looks more like this though:

Code: [Select]

java.lang.Exception: InvalidOperationException : Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
       at System.Collections.Generic.Dictionary`2[TKey,TValue].FindEntry(System.Collections.Generic.Dictionary`2[TKey,TValue])
       at System.Collections.Generic.Dictionary`2[TKey,TValue].TryGetValue(System.Collections.Generic.Dictionary`2[TKey,TValue])
       at HutongGames.PlayMaker.ActionData.GetActionTypeHashCode(HutongGames.PlayMaker.ActionData)
       at HutongGames.PlayMaker.ActionData.CreateAction(HutongGames.PlayMaker.ActionData)
       at HutongGames.PlayMaker.ActionData.LoadActions(HutongGames.PlayMaker.ActionData)
       at HutongGames.PlayMaker.FsmState.get_Actions(HutongGames.PlayMaker.FsmState)
       at HutongGames.PlayMaker.Fsm.Preprocess(HutongGames.PlayMaker.Fsm)
       at HutongGames.PlayMaker.Fsm.Init(HutongGames.PlayMaker.Fsm)
       at PlayMakerFSM.Init(PlayMakerFSM)
       at UnityEngine.Object.Internal_InstantiateSingle(UnityEngine.Object)
       at UnityEngine.Object.Instantiate(UnityEngine.Object)

We're using Playmaker 1.9.6.f1 with Unity 2021.3.23f1.