playMaker

Author Topic: Performance Tests  (Read 406 times)

DanielThomas

  • Beta Group
  • Full Member
  • *
  • Posts: 148
    • View Profile
Performance Tests
« on: December 20, 2019, 07:59:01 AM »
Hi, so I was curious about the performance impact using Playmaker vs regular script which led me to do a couple of tests. While not exact, for me, they can still be a general guide to when planning the design of the FSMs.

The test was made in an empty scene and in profiler targeted to a build. They're an estimate of the lowest and highest ms of the Update.ScriptRunBehaviourUpdate.

The scene has 200 game objects with 2 FSMs each  (400FSM in total).

Cost of Playmaker FSM itself:
How much is the overhead cost of just having a running FSM?
-  The FSM states are doing nothing. Script is a new script (empty Start() and Update())
  • No FSM enabled 0.01 - 0.06ms
  • 400 FSMs Enaled 0.18 - 0.68ms
  • No FSMs, 400 scripts enabled 0.02 - 0.07ms
Conclusion: PlaymakerFSM has an overhead cost, scripts doesn't


Turning FSM on and off
So if there is a cost to just having FSMs running, it should be more effective turning them off when not used?
- One of the two FSMs(200 in total) is enabling and disabling the other FSM
  • FSM turning on and off every frame: 0.75 - 2.82ms
  • Same as above, 1 sec interval:         0.21 - 2.75ms
Conclusion: There is a cost the moment FSM is enabled and disabled, while turning 400 on and off at the same time is extreme case, might not be a good idea to do with a lot of FSMs used very frequently.


Loops
How about the loops?
Both FSMs have a state with "int add" action. One scenario with "every frame" checked, the other scenario with a "next frame event" action that loop back to the same event.
  • Every frame ticked:   0.24 - 1.64ms
  • Next frame event:   0.57 - 4.16ms
Conclusion: This was a bit surprising as the result feels significant. So at the moment, it feels using "every frame" is more performant than using the next frame to create a loop(usually when using action sequence).


Broadcast All
How about using broadcast all?
- One FSM that broadcast all event to the 400 FSMs, or a script doing the same.
  • Broadcast from FSM: 0.09 - 0.28ms
  • Broadcast from script: 0.07 - 0.32ms
Conclusion: "broadcast all" is the same from a script and an FSM.


Final Thoughts:
While the tests aren't exact, I think they show a relative cost between the scenarios put against each other.
- I've had projects where I had FSMs to environment objects such as trees. Objects that you usually have a couple of hundreds in the scene.
So my conclusion would lead me to believe, even though the FSMs are doing nothing there is still a big cost to have them running. My previous solution was to disable the FSMs, colliders, and visuals when the object goes out of the camera. So this would be a balance between the cost of having them all running and the cost of constantly turning them on and off. Best would of course to have a manager FSM that did everything for the objects when needed. Or activate them when needed through collision etc.
- The result would also indicate that I should use "every frame" as much as possible, especially with FSMs in a constant loop.
- I would use scripts as much as possible, especially for small things that just have a small responsibility (is I would be comfortable writing this in a script) to decrease the number of FSMs running.

I would love to get more insight into this, maybe there are other things as garbage collection(which I don't understand much about) that I'm missing.
« Last Edit: December 25, 2019, 01:04:29 PM by DanielThomas »

daniellogin

  • Full Member
  • ***
  • Posts: 199
    • View Profile
Re: Performance Tests
« Reply #1 on: December 20, 2019, 08:31:58 PM »
Thanks. This is pretty interesting.

While I'll be trying to consider all of this, the main one I think is that an idle FSM is not free.

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 15082
  • Official Playmaker Support
    • View Profile
Re: Performance Tests
« Reply #2 on: December 23, 2019, 11:49:29 PM »
Hi,

 Well, it is interested in theory.

-- did you test turning on and off 400 scripts component?
-- your loop test gives really odd results indeed. but really you should never use nextframe event for looping! 100 items will take 2 seconds to process at 50FPS, this is not acceptable. I think you should test these with 100 fsm doing the looping, not just one fsm, and then profile usage overall.

For testing playmaker against script in a real project, I ported 100% Unity 2d platformer, which is a fully playable gameplay with lots happening, and the results I got are here:

https://github.com/jeanfabre/PlayMaker--UnityLearn--2dPlatformer

I think it reflects more the difference between playmaker and scripts, because it's used in a real environment and on different platform.


Bye,

 Jean


DanielThomas

  • Beta Group
  • Full Member
  • *
  • Posts: 148
    • View Profile
Re: Performance Tests
« Reply #3 on: December 26, 2019, 05:31:12 AM »
Jean,

Quote
-- did you test turning on and off 400 scripts component?
I didn't as it didn't feel as it would be needed considering how little performance impact an "idle" script was taking.

Quote
-- your loop test gives really odd results indeed. but really you should never use nextframe event for looping! 100 items will take 2 seconds to process at 50FPS, this is not acceptable. I think you should test these with 100 fsm doing the looping, not just one fsm, and then profile usage overall.
I think I was unclear, all the FSMs was doing the looping. All tests were with the 200objects and two FSM's on each. I don't know why you shouldn't _ever_ use the nextframe for looping. Let's say you need a variable to change in a very specific order before you use it, and that every frame. It would make sense to use an action sequence and next frame loop.
But I'm actually more curious the reason it has such an impact in the first place, as the action itself isn't performant heavy?

Code: [Select]
For testing playmaker against script in a real project, I ported 100% Unity 2d platformer, which is a fully playable gameplay with lots happening, and the results I got are here:

https://github.com/jeanfabre/PlayMaker--UnityLearn--2dPlatformer

I think it reflects more the difference between playmaker and scripts, because it's used in a real environment and on different platform.
I know about the test, which also was the reason I was under the impressions Playmaker doesn't have any perfomance impact (unless you use get/set). My testing is more mather of fact which would help keeping good performance. If you have 100 units with a lot of FSMs (to split up by responsibility, which is good practice from what I understand) you probably would want to rethink that in to some kind of system where you have one manager. Even having FSM's for holding data can have a significant impact when you start to come up in the hundreds. Either use a script or have the manager have the data and store any unit that get stats change.

But I also understand it's a use case, only a few projects will have a lot of FSMs on the scene at the same time - it was just something for me to keep in mind when planing my project architecture.

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 15082
  • Official Playmaker Support
    • View Profile
Re: Performance Tests
« Reply #4 on: December 27, 2019, 02:37:23 AM »
Hi,

Speed tests make sense when you overload them, because on a singular use of an api or action, a few milliseconds more or less really doesn't matter at all. It's in your game loop that it matters.

I think your tests are biased with some underlying overhead which blur the results.

I never use next frame event for loops, it's like taking a coffe break every time you enter a a new entry in an excel document. that doesn't fit the design patterns of processing as fast as possible a bunch of data. I use next frame event ONLY when the sequence matters and I have several Fsm acting together to provide a features and they need to wait completion of other fsm, then  making use of next frame event is totally making sense. I probably understand where you come from with this, because of the visual nature of FSM, you tend to want to give time for things to process visually, but when it's all compiled for your game, it doesn't matter anymore.

Actually, any complex projects will have hundreds of fsm running at the same time, so the use case of having hundreds of fsm doing the same thing and check perfs makes more sense then testing two single fsm doing a variant of the same features.

Bye,

 Jean