playMaker

Author Topic: Touch input lag  (Read 8549 times)

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Touch input lag
« on: May 01, 2018, 02:33:09 PM »

I'm working on a simple 2D game. When I mean simple, it is really that simple.
It testes the player's reflexes.

I work on Unity 2017.3.0f3 and Playmaker 1.8.8.f2.

I needed to see how far I could push the touch inputs and outputs in real time on some current and slightly older Android devices, so I made a simple project for that.
It contains nothing else but Playmaker, Ecosystem and lately I just added uGUI but the problem I spotted existed before that anyway.

I've built a multi-touch system with FSMs only and it works perfectly. I'm not testing on Unity Remote 5, it's useless. I build apks.

I have FSMs devoted to printing stuff on the screen for testing purposes, in real time, such as coordinates, distances, framerate, etc.
Also I have two UI images (gizmos) that are displayed under the fingers.
They appear when I press the screen and get hidden when I left the finger (phase: ended).
The main problem is that there's a huge lag. Not only as the gizmos trail behind the fingers (when I make circles very fast I can almost catch them up with my fingers!) but also when pressing. It's obvious that the input isn't registered fast enough.
The moves themselves are smooth, I run at near 60 FPS; but really, the gizmos I use don't follow the fingers in true real time, they clearly suffer terrible lag. I'd say, typically 150 - 250 ms, it really shows.
And it's absolutely abysmal for the game I'm making.

I made the scripts as simple as possible. The data of each finger input is checked every frame in its own FSM.
Typically, I have 3 FSMs for each finger:

1. State : manages the flux between the states according to the inputs and their phases.
2. Data : one state FSM that collects touch info in real time.
3. Display : manages the showing/hiding of each gizmo, plus its positionning.

The first FSM is almost an enhanced version of the FSM shown in the official tutorials, save for the fact that the starter state can also trigger a "moved" since I noticed that starting with a swipe can get missed by the "touched" phase.
Regarding the third FSM; I tried several ways to place the gizmo, direct "set property" actions (by dragging the finger game object into the state tab) on the rect transform, fiddled with other positionning actions, but they all have the same problem. The lag remains.

Everything else in the build is meant to be set to low quality. No complex lighting, the project is almost pure vanilla.

I tried to find solutions on internet and it turns out that many developers have had this lag issue too. It also turns out that not many devs seem to make games which require rather tight controls.

The interesting point is that someone I knew coded a simple game for Android, one that needed responsive controls and there never was any kind of lag of that nature. It was pure c#.


Why I think Playmaker MAY be the culprit will be explained below.


https://forum.unity.com/threads/touch-input-lag-other-games-are-fine.239253/
https://forum.unity.com/threads/touch-lag-help.483178/
https://answers.unity.com/questions/1130946/android-touch-input-lag-1.html
https://forum.unity.com/threads/input-lag-android.372943/

They have this problem and the same lag I measured.
No solution was ever provided in those threads.


https://forum.unity.com/threads/how-to-defeat-latency-on-touches.93365/

Now that's a more interesting thread.
It ends with a (shameless :)) link to a plugin (again, another one, *sigh*) that calls the input at a native level, sidestepping Unituh.
https://assetstore.unity.com/packages/tools/input-management/ios-native-touch-107065
Good... if you know how to code.
And that's for iOS only.


https://forum.unity.com/threads/unity-ui-dragging-touch-input-feels-pretty-slow-on-android-compared-to-other-games.470536/
Unity UI Dragging (touch input) feels pretty slow on Android compared to other games?

Now this one is interesting.
User dandepeched posted a video and then also said that:

Quote
I've also had performance problems with Touch events on Android. I didn't find any solution on forum so I was thinking about proper way to implement Touch and Mouse.
I do find very good video on this, maybe it can help to someone -


But this does not fix my problem, so I start to analyze what can make such lag. I do found the root cause - external library. I was using "Outline Effect" plugin from Asset store. It was introducing huge lag on Android. After disabling it everything goes really smooth. So you should consider asset store stuff also.

Hope this tips can help someone.


Is there something similar in Playmaker that could be causing the problem?
If it cannot be solved, I literally may have to jump to another engine, a 2D one that already includes visual scripting by default (there are a couple very powerful ones).


Also, right under Scripting Define Symbols in Android Player Player / Configuration, there's a section that's been added around 2017.1 I believe, Active Input Handling although
Looks like this option isn't displayed by default, it's not even documented on Unity's website (https://docs.unity3d.com/Manual/class-PlayerSettingsAndroid.html), as they literally forgot to explain it.
So I wonder if it's related to the symbols. Here's those I have:
PLAYMAKER;PLAYMAKER_1_8;PLAYMAKER_1_8_8;PLAYMAKER_1_8_OR_NEWER;PLAYMAKER_1_8_5_OR_NEWER;ECOSYSTEM;ECOSYSTEM_0_6;ECOSYSTEM_0_6_OR_NEWER;PLAYMAKER_UTILS

Going back to the idea that there could be a problem with a library or something, could some Playmaker component be the source of the trouble?


djaydino

  • Administrator
  • Hero Member
  • *****
  • Posts: 7615
    • jinxtergames
Re: Touch input lag
« Reply #1 on: May 01, 2018, 05:36:58 PM »
Hi,
These symbols are there to check if you have playmaker installed (same for the Ecosystem) the should not have anything to do with performance.

I think the issue is a unity issue and the playmaker touch actions uses the standard (unity) coding.

But i can't really confirm if its laggy as i use 3rd party assets for touch a looong time already and never used the standard (unity)

Also keep in mind that the devices touch always have some lag ( you can test this on the device, somewhere in the developer options you can set 'Show touches')

Can you show a video on how much lag it is and maybe pm me a link to your project so i could test (with pm actions/in C#/3rd party asset) ?

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 15500
  • Official Playmaker Support
Re: Touch input lag
« Reply #2 on: May 02, 2018, 03:26:53 AM »
Hi,

 PlayMaker has nothing to do with touch input lags, you should very careful monitor your game on the device using the unity profiler, and check the framerate and especially what happens whenb the user touches the screen, it may be responding fine but what you want Unity to do is taking time and this is the lga you witness.

 so to test this, make a totally simple button or cube with a touch trigger that change color of the background randomly or something, and see how fast it reacts, than compare this with your game with all your features, gameplay, etc etc, if there is a difference, it's your game logic or graphics or sounds that are responsible.

 Bye,

 Jean

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Touch input lag
« Reply #3 on: May 02, 2018, 08:35:09 AM »

Hello,

Yes, the actions use the touch.input etc. There shouldn't be any problem.
The native option for showing touches on an Android device did indeed reveal a slight lag on the phone's side, but it was acceptable.
This latency is not the one found in Unity.
It looks like this "natural" latency gets multiplied by Unity.

I'll not integrate my touch system into the game yet if I can't even have it work properly in a test project.
It's weird because it's absolutely smooth, the frame rate is totally ok and every single input I generate by touching the screen (touch or swipe) is properly memorized. There's no loss of data.
It's just that all the stuff data arrives many frames later.
If there was a performance bottleneck of some kind, the frame rate would simply fall to its knees as long as I'd be touching the screen. Such is not the case.

Also, I have a rotating cube that does change colour. It works but the latency is present too: the colour is changed with a noticeable delay which just happens to be that of the latency itself.
Clearly, any Unity code that uses any input's info/data gets treated with a considerable delay and yet the FPS are high.

Every single input is kind of affected by an "input saved! see you x frames later" "please hold the line" phenomenon.

I'll run those tests, if the profiler is nice and spots my android device (does not always work).

On a sidenote, this time I got a warning during the building process, which I aborted:

Quote
Game scripts or other custom code contains OnMouse_ event handlers. Presence of such handlers might impact performance on handheld devices.
UnityEditor.BuildPlayerWindow:BuildPlayerAndRun()

I was allowing the cube's colour to be modified with a mouse click. Been removed but the problem remains.

Additionally, I had noticed having few but regular global illumination spikes in the profiler (around 15-20 ms) despite having low quality settings, no calculations, shadows, no light, no nothing. :)
Crazy stuff.

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Touch input lag
« Reply #4 on: May 02, 2018, 09:52:38 AM »
Just curious but do I need to pay for Unity Plus in order to be able to use the Profiler with the apk running on the smartphone?

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Touch input lag
« Reply #5 on: May 02, 2018, 01:56:14 PM »
There are days when things decide to act in a wonky fashion, so much that you have doubts about procedures you took for granted, and you end wondering if you need U+ to have access to the Profiler (a reminder of the old days I guess) because all logic has flown through the window for 24 hours and the computer is just being a nuisance. :D
That wretched Profiler wouldn't start (regardless of how much I'd configure the settings and even go grab adb's data and so on) and then poof! it wen't on after launching the app like half a dozen times. Oh and the building process decided it was time to make my phone crash, just for the sake of it, by producing a flawed apk. Because otherwise things would be too easy, right? :/

So, I got my hands on several Android devices and the issue was found on all of them. Definitely, the final latency was clearly the result of the native latency being multiplied by some unknown factor when going through the Unity grinder.

Anyhow, now things work.
I made a few modifications. Most notably, I pushed all touch data into a normalized state, so I then had to multiply the fingers' x and y by the screen's resolution.
Now the latency is very tolerable, all is extremely responsive (touches and swipes) and the gizmos drawn by Unity almost stick in real time to the traces drawn by the Android system on the phone.
:)

I'd say, by all means, use normalized data when collecting info from your touch inputs, it seems to cut the latency for some odd reason.

I'm weary of saying the problem is solved but if I don't post anything else in the coming days then it will be safe to say it's not a problem anymore.

PS : djaydino, I hesitated using Easy Touch 5 but the documentation is terrible, the  add-on weighs more than 11 mb and my project is already getting too fat (and I have to add advertising SDKs), plus I don't need all the special controllers and couldn't be bothered to check what I could keep and remove form the assets.

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Touch input lag
« Reply #6 on: May 03, 2018, 03:25:40 AM »
Another problem of separating major activities into separate FSMs (check the order I described in my first post) is that if they depend on each other to some degree, I don't know how to have them operate in order, within the same frame.

Having the gathering of inputs' information be managed in its own FSM could also explain that "Get touch info" actually happens too soon in the frame, meaning it's collecting the data from the former frame (n-1).
And to make matters worse, it's quite possible that the graphics (the gizmos that represent the fingers' respective positions, coded in Playmaker) may even display data from the frame n-2 (haven't found a way to force them in a late update).

All of these actions run with the "every frame" tag on. I wish I knew how to solve that issue without having to put ALL the actions back into not only one FSM, but actually one single, cramped state.

Otherwise, I would have to first test touch phases (began, ended, etc.) in one state, then move on to another state that gets the data, then again another state that does the gizmo update, and finish all that with a Next frame event action to make all of this loop. All of this without the "every frame" tag for any of the actions.
But I'm a bit afraid of missing some inputs this way.

However, this may reduce the latency on Unity's side even more, although I must say, some Android devices that look good actually have pretty crappy response times!

Why is the native input latency not a parameter taken into account for phone specs? It's very important and makes the use of a device more confortable when it responds quickly.

To get back to the topic, I noticed a "manual update" option in the PM Inspector's FSM tab.
It's said that one can even call this manual update through "PlayMakerFSM.Fsm.Update"

I suppose that this could allow me to order the execution of FSMs within a frame, right?

Looks a bit more advanced than the usual PlayMaker way of doing things. Is it possible or even wise to try to use that through an action, like "Invoke" or "Call static method" or something? What would be the proper syntax to use this function?
« Last Edit: May 03, 2018, 03:29:28 AM by Broken Stylus »

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 15500
  • Official Playmaker Support
Re: Touch input lag
« Reply #7 on: May 07, 2018, 02:28:02 AM »
Hi,

 I think you are over complicating things. Instead let's do it another way.

 Can you explain what is the feature you are trying to achieve? and what is it that you are splitting across different fsm, yet with common data that need to be processed sequentially.

also, I think you are likely misunderstanding from where touch latency comes from, again, without profiling ( it's hard I give you that...), it's impossible to say what's responsible for this kind of behaviour.

manual update would not solve your issue.


Your issues are typical, so I think it's important to find the right solution for this. Once you will pick up how to do this, you'll be set.


Bye,

 Jean

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Touch input lag
« Reply #8 on: May 07, 2018, 10:17:41 AM »
A request... OK.

If there is one thing I could need and that's why I was looking at a FSM's manual update, although I admit it's more of a curiosity, it is a way to force any kind of action into a Late Update.
However, actions not created with that feature don't provide that option so I'm wondering if there's a way around that.

This is why I went looking into the FSM's option and spotted the "manual update" thing. I thought ah ah! Maybe I could tell my "graphics" FSM to wait and get active at the end of the frame, just to be sure that the data it's going to use to place the gizmo(s) will be fresh and not one frame late.

As a reminder, typically, in my 3-FSMs structure, the third one is dedicated to graphics and it would be nice to force it to be dealt with at the end of a frame, because when putting "Get Touch Info" in every frame mode, in one FSM, and putting gizmo management (display + position) in another (every frame mode too), I simply cannot know which FSM is actually doing its code first.

When I first got into Playmaker I quickly figured that it would be a good thing to have FSMs run parallelal to each other when possible, some sort of coroutine simulation I guess.
But without knowing when each piece of visual scripting gets managed first, there are times when it is actually better to put all your eggs in the same basket... or FSM.
Sidenote: would there be a way to actually know that kind of things with the FSM Timeline? That said, the info would only be useful if the user could control the order in which FSMs run their codes during a frame.

This is why I fusioned my last two FSMs into one (data + graphics), following the logic described two posts earlier: 1. grab data, 2. do displayed stuff, 3. wait for next frame (using Next Frame event action).
This works very well.

I consider the problem I experienced largely solved, especially since there are incompressible things such as the device's native tactile latency which is just plain bad sometimes; and again, some crapfones gave me better results than quite expensive ones.

So yes, really, the one thing I'm curious about now is how to use the Manual Update thing, because to me it sounds like a way to decide when to run a FSM's code during a frame, but maybe I'm wrong?
What I read here (http://hutonggames.com/playmakerforum/index.php?topic=15182.0) looks very interesting. I went looking on Ecosystem but didn't find an action that can manage and control the update moment of another FSM.
Especially that interesting "FSM Update" action Alex Chouls talked about, perhaps one with the capacity to precisely aim for uGUI related Updates : the action would let one chose the type of update, that is, when the FSM will be read and its code executed. The drop down menu would include the list of all official Unity types of updates, including those dedicated to specific types of game objects or components.
Perhaps with a number (integer) for the order too, in case this is used for several FSMs ? This would be optional, because one way to "order" several FSMs' updates would simply be to duplicate the same action inside one state and be super strict by using the "action sequence" option.
Or the action could ask how many FSMs we want to manage (something similar to the action "Build string" that asks how many elements the action is going to glue together), with a "strict order" option?
« Last Edit: May 07, 2018, 10:24:04 AM by Broken Stylus »

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Touch input lag
« Reply #9 on: May 07, 2018, 10:20:53 AM »
Also, on the topic of performance and manual updates:
https://forum.unity.com/threads/efficient-updating-take-control-of-update-functions.479509

Quote
In this tutorial we will cover how to bypass the Unity Update() functions to be less resource-intensive and perform faster. Studies show that handling Updates yourself is 4X or more faster than letting Unity handle them.

Four times faster or more? Could this have an effect on the latency inside Unity's player?

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 15500
  • Official Playmaker Support
Re: Touch input lag
« Reply #10 on: May 09, 2018, 04:12:08 AM »
Hi,

 I think you are overthinking this. I'd to be able to repro this before we go down some crazy setup, I never used the manual update. Let me play around with it and I'll get back to you on this.

Can you explain the feature you a retrying to do and I'll see if I can do this and provide the right easiest way to go about this.

have you experimented with running fsm as templates inside your state? then theay would be ran in order they are defined.

 Bye,

 Jean

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Touch input lag
« Reply #11 on: May 09, 2018, 06:36:02 AM »
Hi,

Yes, I have experimented with running FSMs.
Unless I get it wrong, "Run FSM" is creating and destroying FSM "objects" on the fly, thus polluting the memory quicker and forcing more garbage collections if used repeatedly, right?

Pooling FSMs also seems redundant instead of simply creating my "display" FSM as a component on the gizmo.
At best, I'd be switching this FSM on and off with "Enable FSM" from another FSM (I already used that on/off methodology multiple times and it's okay).

The only problem with this action (and "Run FSM" too) is that it does not allow the user to tell when to run the code of the FSM by exploiting the inbuilt update phases of Unity.

Now, thinking of it, would it be possible to actually have an option in the FSM that simply forces the whole FSM to be ran during a specific update phase?

What I mean is that instead of, or next to "manual update", one could tick or select an option from a drop down menu to force the entire FSM and ALL of its contents to run at the selected time (update phase), using Unity player's native cycles.
Late Update, Fixed Update, etc.

That way, all actions would see their update time aligned to that of their parent FSM and allow the user to create very specific FSMs.
Now I don't know if that's doable, to force an action to run at "late update" for example when it's actually typed with voidUpdate...
 
At the moment, without using the very few actions that have the "late update" option, it's impossible (I think) to create a very late updatish FSM.

As said earlier, I can control when a FSM will be run, but it will run during the normal "update" phase nonetheless; unless it contains an action with a "late update" method or option selected but that's most like an exception and anyways, as I said, such actions are a rarity.

I don't think it's possible yet to create a whole FSM with "normal" actions and then sort of convert the entire FSM to have it run at "late update".
Like errr... some kind of mod, a "wrapper", see?
That would be neat for example if making a (large) FSM completely devoted to graphics.

Of course it removes the flexibility of the action described in a former post, because once this FSM is defined as to update outside of the normal update phase, it will be that way forever.
But is that flexibility needed?
Typically, when you create your FSM, you do know when you expect it to run anyways, you're not going to change its execution time on the fly to move it up or down the overall execution cycle, like from "update" to "late update", during your game. That would be silly.

So yes, I think setting the mode on the FSM itself would be a good thing.
Again, I create the FSM and then select "late update" and poof, it will be forced to run during said "late update" phase, regardless of the actions' complaints. >:)
It's magic!

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 15500
  • Official Playmaker Support
Re: Touch input lag
« Reply #12 on: May 11, 2018, 02:07:26 AM »
Hi,

 Being able to define the order of execution of individual fsm would be cool indeed. but again, I am pretty sure you can get away without this in 99% of the cases, which is why I think it would be good that you explain exactly what you want to achieve so that I can bump into the same problem and find a solution.

 Bye,

 Jean

Broken Stylus

  • Beta Group
  • Hero Member
  • *
  • Posts: 772
Re: Touch input lag
« Reply #13 on: May 11, 2018, 11:10:23 AM »
Hi,

 Being able to define the order of execution of individual fsm would be cool indeed. but again, I am pretty sure you can get away without this in 99% of the cases, which is why I think it would be good that you explain exactly what you want to achieve so that I can bump into the same problem and find a solution.

 Bye,

 Jean

Yes indeed, it's kinda very technical but I find myself easily turning into a frame rate nazi.  ;D

I may have noticed a slight difference in speed between using PlayMaker uGUI proxy components (onClick or others) and directly using Unity's "Trigger Event" component + PlayMakerFSM.SendEvent.
Both types of components were placed on the same object, a uGUI button, and then their events sent to the same FSM, each event triggering its own respective state.
I noticed that the FSM had time to be triggered by the events sent by Unity's "Trigger Event" component just before the events sent by the PlayMaker proxies would trigger their own respective states.
So Trigger Event seems faster.

Other than that, I consider the problem solved because what I suggested in the final moments of that thread could only be achieved through real coding today, and I've reached a point I'm okay with the performances I obtain.
The rest is up to Unity's own latency in receiving data from the smartphone and said device's own native latency.

The thread drifted towards a discussion about a general idea that came to me realizing what would be cool, i.e., pushing entire FSMs into specific update phases so I don't have to build extremely contrived systems where, for example, a FSM dedicated to graphics would have to wait for events sent by a whole lot of other FSMs telling it that they've finished doing their technical stuff and now visuals can be updated.