playMaker

Author Topic: Performance Tests  (Read 4174 times)

DanielThomas

  • Beta Group
  • Full Member
  • *
  • Posts: 150
Performance Tests
« on: December 20, 2019, 09:59:01 AM »
Hi, so I was curious about the performance impact using Playmaker vs regular script which led me to do a couple of tests. While not exact, for me, they can still be a general guide to when planning the design of the FSMs.

The test was made in an empty scene and in profiler targeted to a build. They're an estimate of the lowest and highest ms of the Update.ScriptRunBehaviourUpdate.

The scene has 200 game objects with 2 FSMs each  (400FSM in total).

Cost of Playmaker FSM itself:
How much is the overhead cost of just having a running FSM?
-  The FSM states are doing nothing. Script is a new script (empty Start() and Update())
  • No FSM enabled 0.01 - 0.06ms
  • 400 FSMs Enaled 0.18 - 0.68ms
  • No FSMs, 400 scripts enabled 0.02 - 0.07ms
Conclusion: PlaymakerFSM has an overhead cost, scripts doesn't


Turning FSM on and off
So if there is a cost to just having FSMs running, it should be more effective turning them off when not used?
- One of the two FSMs(200 in total) is enabling and disabling the other FSM
  • FSM turning on and off every frame: 0.75 - 2.82ms
  • Same as above, 1 sec interval:         0.21 - 2.75ms
Conclusion: There is a cost the moment FSM is enabled and disabled, while turning 400 on and off at the same time is extreme case, might not be a good idea to do with a lot of FSMs used very frequently.


Loops
How about the loops?
Both FSMs have a state with "int add" action. One scenario with "every frame" checked, the other scenario with a "next frame event" action that loop back to the same event.
  • Every frame ticked:   0.24 - 1.64ms
  • Next frame event:   0.57 - 4.16ms
Conclusion: This was a bit surprising as the result feels significant. So at the moment, it feels using "every frame" is more performant than using the next frame to create a loop(usually when using action sequence).


Broadcast All
How about using broadcast all?
- One FSM that broadcast all event to the 400 FSMs, or a script doing the same.
  • Broadcast from FSM: 0.09 - 0.28ms
  • Broadcast from script: 0.07 - 0.32ms
Conclusion: "broadcast all" is the same from a script and an FSM.


Final Thoughts:
While the tests aren't exact, I think they show a relative cost between the scenarios put against each other.
- I've had projects where I had FSMs to environment objects such as trees. Objects that you usually have a couple of hundreds in the scene.
So my conclusion would lead me to believe, even though the FSMs are doing nothing there is still a big cost to have them running. My previous solution was to disable the FSMs, colliders, and visuals when the object goes out of the camera. So this would be a balance between the cost of having them all running and the cost of constantly turning them on and off. Best would of course to have a manager FSM that did everything for the objects when needed. Or activate them when needed through collision etc.
- The result would also indicate that I should use "every frame" as much as possible, especially with FSMs in a constant loop.
- I would use scripts as much as possible, especially for small things that just have a small responsibility (is I would be comfortable writing this in a script) to decrease the number of FSMs running.

I would love to get more insight into this, maybe there are other things as garbage collection(which I don't understand much about) that I'm missing.
« Last Edit: December 25, 2019, 03:04:29 PM by DanielThomas »

daniellogin

  • Full Member
  • ***
  • Posts: 215
Re: Performance Tests
« Reply #1 on: December 20, 2019, 10:31:58 PM »
Thanks. This is pretty interesting.

While I'll be trying to consider all of this, the main one I think is that an idle FSM is not free.

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 15500
  • Official Playmaker Support
Re: Performance Tests
« Reply #2 on: December 24, 2019, 01:49:29 AM »
Hi,

 Well, it is interested in theory.

-- did you test turning on and off 400 scripts component?
-- your loop test gives really odd results indeed. but really you should never use nextframe event for looping! 100 items will take 2 seconds to process at 50FPS, this is not acceptable. I think you should test these with 100 fsm doing the looping, not just one fsm, and then profile usage overall.

For testing playmaker against script in a real project, I ported 100% Unity 2d platformer, which is a fully playable gameplay with lots happening, and the results I got are here:

https://github.com/jeanfabre/PlayMaker--UnityLearn--2dPlatformer

I think it reflects more the difference between playmaker and scripts, because it's used in a real environment and on different platform.


Bye,

 Jean


DanielThomas

  • Beta Group
  • Full Member
  • *
  • Posts: 150
Re: Performance Tests
« Reply #3 on: December 26, 2019, 07:31:12 AM »
Jean,

Quote
-- did you test turning on and off 400 scripts component?
I didn't as it didn't feel as it would be needed considering how little performance impact an "idle" script was taking.

Quote
-- your loop test gives really odd results indeed. but really you should never use nextframe event for looping! 100 items will take 2 seconds to process at 50FPS, this is not acceptable. I think you should test these with 100 fsm doing the looping, not just one fsm, and then profile usage overall.
I think I was unclear, all the FSMs was doing the looping. All tests were with the 200objects and two FSM's on each. I don't know why you shouldn't _ever_ use the nextframe for looping. Let's say you need a variable to change in a very specific order before you use it, and that every frame. It would make sense to use an action sequence and next frame loop.
But I'm actually more curious the reason it has such an impact in the first place, as the action itself isn't performant heavy?

Code: [Select]
For testing playmaker against script in a real project, I ported 100% Unity 2d platformer, which is a fully playable gameplay with lots happening, and the results I got are here:

https://github.com/jeanfabre/PlayMaker--UnityLearn--2dPlatformer

I think it reflects more the difference between playmaker and scripts, because it's used in a real environment and on different platform.
I know about the test, which also was the reason I was under the impressions Playmaker doesn't have any perfomance impact (unless you use get/set). My testing is more mather of fact which would help keeping good performance. If you have 100 units with a lot of FSMs (to split up by responsibility, which is good practice from what I understand) you probably would want to rethink that in to some kind of system where you have one manager. Even having FSM's for holding data can have a significant impact when you start to come up in the hundreds. Either use a script or have the manager have the data and store any unit that get stats change.

But I also understand it's a use case, only a few projects will have a lot of FSMs on the scene at the same time - it was just something for me to keep in mind when planing my project architecture.

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 15500
  • Official Playmaker Support
Re: Performance Tests
« Reply #4 on: December 27, 2019, 04:37:23 AM »
Hi,

Speed tests make sense when you overload them, because on a singular use of an api or action, a few milliseconds more or less really doesn't matter at all. It's in your game loop that it matters.

I think your tests are biased with some underlying overhead which blur the results.

I never use next frame event for loops, it's like taking a coffe break every time you enter a a new entry in an excel document. that doesn't fit the design patterns of processing as fast as possible a bunch of data. I use next frame event ONLY when the sequence matters and I have several Fsm acting together to provide a features and they need to wait completion of other fsm, then  making use of next frame event is totally making sense. I probably understand where you come from with this, because of the visual nature of FSM, you tend to want to give time for things to process visually, but when it's all compiled for your game, it doesn't matter anymore.

Actually, any complex projects will have hundreds of fsm running at the same time, so the use case of having hundreds of fsm doing the same thing and check perfs makes more sense then testing two single fsm doing a variant of the same features.

Bye,

 Jean

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Performance Tests
« Reply #5 on: April 20, 2020, 12:18:35 PM »
Cost of Playmaker FSM itself:
How much is the overhead cost of just having a running FSM?
-  The FSM states are doing nothing. Script is a new script (empty Start() and Update())
  • No FSM enabled 0.01 - 0.06ms
  • 400 FSMs Enaled 0.18 - 0.68ms
  • No FSMs, 400 scripts enabled 0.02 - 0.07ms
Conclusion: PlaymakerFSM has an overhead cost, scripts doesn't
Without knowing what is going in Playmaker's DLLs, I suppose that it's the cost of PM checking for every single FSM during a frame.
0.68 ms is absolutely negligible. Less than 1 ms on a total of 400 FSMs doing nothing is not a big problem at all. That's literally less than a sixteenth of a frame at 60 FPS (1 frame = 16.667 ms).

PolyMad

  • Hero Member
  • *****
  • Posts: 545
Re: Performance Tests
« Reply #6 on: April 20, 2020, 01:22:01 PM »
Just for the lulz: I once added one FSM to a rock that was procedurally instanced on a terrain in the number of hundreds of times.
The FSMs were supposed to remove the rigidbody component once the rocks were sleeping and then kill itself.
It was working quite fine, so I forgot about it.
Then things got serious, and I added something like 20000 of these rocks.
Needless to say, it froze the hell off and took it some minutes to calculate the whole physics and kill the FSMs. Not less than 3 minutes.
But I guess a big part in there was the physics engine.

I did my few tests too with Playmaker in the past and I think that it's perfectly viable up to about 500-1000 components, depending of course on the complexity and how well the FSMs are designed and called.
This is perfectly fine for A LOT of casual and non-casual games.
It only can't compete with games that require huge performance, like hundreds of soldiers on screen and a lot of other stuff.
But until you are below 100 enemies and moving elements, I don't think that Playmaker will have any significant impact on the project.

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Performance Tests
« Reply #7 on: April 28, 2020, 07:05:25 AM »
The FSMs were supposed to remove the rigidbody component once the rocks were sleeping and then kill itself.

Have you tried using the  System event DISABLE, just to see what happens?
This event is a bit tricky to use since certain actions (not playmaker actions) will be ignored, perhaps because too long to process during this final special frame that is triggered by the DISABLE event. The order of components on a game object does not seem to be an issue, although the quantity and complexity might. Many things can be done to the Transform during the "DISABLE frame" but one should not do actions that depend on other objects, like (re)parenting (unless you put "null").
The DISABLE frame is fairly lax when it comes to the number of loops you can go through before the end of it, but you should not have anything system-time related (like a 'Wait' Playmaker action). You must do instantaneous things during that frame.
It's some kind of special "late update" and once components have been dealt with, they won't be considered twice: I tried to create a logic paradox but Unity avoided it because the object is put in a unique transition state from active to inactive, whether it's done by deactivating the GO or deleting it, so you cannot relaunch this transition a second time. This is why when you try to do illegal actions during this transition that the console will throw errors like "can't do this since the GO is already being activated or deactivated."

PolyMad

  • Hero Member
  • *****
  • Posts: 545
Re: Performance Tests
« Reply #8 on: April 28, 2020, 09:10:20 AM »
Thank you for the deep insight: it's always useful learning specific stuff like this.
But the thing is that in this case the system had to be centralized.
Or at least, I would work it differently if I had to go through it again:
  • I would make a FSM on manager
  • Identify all the objects with some naming convention or tag
  • Start all the objects sleeping, without gravity
  • Activate their gravity and awake them in chunks so to not overload the physics engine
  • Remove the rigid body
That choice was quicker, and good for few hundreds of objects, even though very wrong under a performance point of view, but it was really extremely stupid to even think to set it working on thousands of objects!  ;D
But I did it also with performance intent. I saw that it was holding well up to few thousands, so I thought "why not"  ;D

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Performance Tests
« Reply #9 on: April 28, 2020, 11:05:21 AM »
Even big games don't push their engines that far, they still use tricks to give the impression of an environment being more dynamic than it truly is.

PolyMad

  • Hero Member
  • *****
  • Posts: 545
Re: Performance Tests
« Reply #10 on: April 28, 2020, 12:24:37 PM »
Ye you have to wake up stuff when the player gets near to it.
But in this case my objective was to create a slightly different world each match, using the physics engine to position the objects like larger rocks, and then make them static.