playMaker

Author Topic: Mouse Pick is WRONG in "Virtual Reality Supported" Mode  (Read 2080 times)

westingtyler

  • Full Member
  • ***
  • Posts: 192
    • View Profile
    • My Oculus Rift + Touch VR Game Using Playmaker
Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« on: February 27, 2017, 11:29:41 PM »
Been living with this bug for like 7 months now, so I figured I'd ask again to see if anyone has any insights. It's the biggest bug in my project and makes game dev miserable.

Problem: Using a Mouse Pick action while "Virtual Reality Supported" (Oculus or OpenVR) causes the Mouse Pick to store an incorrect position on the screen.

Here is a video of the problem in action:
https://www.youtube.com/watch?v=wEiAFg9KpM8

+ Disabling "Virtual Reality Supported" in Unity's Player Settings, ends this problem.
+ Changing the Main Camera's Target Eye from "Both" or "Left" to "None (Main Display)" also ends this problem.

But doing either of those prevents VR from working in my VR game.

+ Increasing the Scale slider at the top of the Game View (higher than 1.0), seems to make the problem worse.

djaydino helped me narrow the problem down but said it was not a playmaker problem and suggested posting to Unity forums, but posting to Unity forums got no response for 7 months, so I am very sad and desperate. I can't possibly believe everyone else using Playmaker and VR also deals with this problem on a daily basis.

This issue exists in Playmaker 1.8.2f7, 1.8.3, and across Unity 5.4 and 5.5, in any combination. This started in 5.4.0f3 when Unity introduced the Scale Slider to Game View and the "Virtual Reality Supported" Player Settings option.

ANY clues would help. Can someone at least confirm that with Virtual Reality Supported (Oculus) they also see the bug? I've attached the package with the scene, which worked fine for djaydino but I do not think he tested in Virtual Reality Supported (Oculus) mode.

Here is the old thread as well. http://hutonggames.com/playmakerforum/index.php?topic=13626.0

Thank you for your time.
« Last Edit: February 27, 2017, 11:34:49 PM by westingtyler »

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 13432
  • Official Playmaker Support
    • View Profile
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #1 on: February 28, 2017, 06:49:23 AM »
Hi,

in VR, you don't have a mouse right? so could you explain what you are after and the use case for your mousepick? you shoudl use a gaze system which means always picking what is at the center of the screen, is that not how you want to select and pick things up?

 bye,

 Jean

westingtyler

  • Full Member
  • ***
  • Posts: 192
    • View Profile
    • My Oculus Rift + Touch VR Game Using Playmaker
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #2 on: March 01, 2017, 09:50:16 PM »
I have support for both a gamepad (and oculus touch controls as soon as I can figure it out) but also for mouse and keyboard. Here is a video showing using the mouse and keyboard in the game, and it works much better than gaze controls for reducing nausea, since you can move your hand without moving your head. Having to look at whatever I want to shoot or pick up, makes me sick.

I use the mouse pick the same as anyone else would, except it casts into 3d virtual screen space. It is hard to explain, but it feels great. Touch controls would be ideal, but until EVERYONE has a touch, I want this option as the main option. Gaze controls are OKAY, but increase nausea for many like myself, because you have to move your head around. https://www.youtube.com/watch?v=_rNf0RBy48k

I already have the Xbox 360 gamepad controls, which centers your hand and uses gaze controls,  but trust me when I say using a Mouse is superior in many ways and preferred. But this means I still need Mouse Pick to work correctly if you have ANY ideas about how to correct this.

To be specific about how this works, your game still shows up on the computer screen in 2d, while you are in the headset, and Mouse Pick works normally on that. It means that the mouse moves when your head moves, but it also means that if you keep your head still you can still move the mouse around the screen, and I have a bounding sphere to keep it a specific distance from the player's face. I cast a mouse pick that hits that invisible spherical barrier around the player, and that is where, in 3d vr space, the hand gets placed.

I appreciate that this sounds weird and foreign, which excited me into thinking I have done something special, because this kicks gaze controls' ass in terms of feel and natural movement.

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 13432
  • Official Playmaker Support
    • View Profile
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #3 on: March 03, 2017, 04:29:13 AM »
Hi,

It could be that you need to build your own raycasting based on mouse screen position, as opposed to the direct mouse pick, likely because you can then defining exactly which camera to use for the raysting. I think the problem is which camera to use for picking things up using the mouse position.

Let me know if that doesn't work.

 Bye,

 Jean

westingtyler

  • Full Member
  • ***
  • Posts: 192
    • View Profile
    • My Oculus Rift + Touch VR Game Using Playmaker
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #4 on: March 05, 2017, 08:31:17 PM »
when you say this, do you mean I need to write my own .cs script? Or can I make this in an FSM? Without Playmaker, I'd have no idea how to make awesome games since I can't code, so if you can offer any insights into or leads about this I will get right on it.

Can I do what you speak of, in playmaker?
« Last Edit: March 08, 2017, 06:00:29 PM by westingtyler »

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 13432
  • Official Playmaker Support
    • View Profile
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #5 on: March 08, 2017, 10:59:06 PM »
Hi,

 no, you have everything you need in PlayMaker, you have lots of raycast actions if you search the action browser, but I think you'll have more success with RayCastFromScreen action that you can find in the Ecosystem.

be careful with screenpoints:
https://docs.unity3d.com/ScriptReference/Camera.ScreenToWorldPoint.html

there is also an action for this: ScreenToWorldPoint

you can also try the action "screenPick"

 Bye,

 Jean

westingtyler

  • Full Member
  • ***
  • Posts: 192
    • View Profile
    • My Oculus Rift + Touch VR Game Using Playmaker
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #6 on: March 13, 2017, 04:04:29 AM »
hey, this is great, and I'll check it out. why should I 'be careful with screenpoints'? what are the dangers of screenpoints?

jeanfabre

  • Administrator
  • Hero Member
  • *****
  • Posts: 13432
  • Official Playmaker Support
    • View Profile
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #7 on: March 15, 2017, 05:38:29 AM »
Hi,

 screen position reference is different then mouse position, y is inverted ( bottom/top). I can't remember which one exactly, but there is a catch here somehow :)

bye,

 Jean

westingtyler

  • Full Member
  • ***
  • Posts: 192
    • View Profile
    • My Oculus Rift + Touch VR Game Using Playmaker
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #8 on: April 02, 2017, 10:42:43 PM »
hi, on the ecosystem I can't find actions called

ScreenToWorldPoint
or
screenPick

i found RayCastFromScreen. With it, I am not sure how to cast the ray from where the mouse is on the screen, how to store the distance, or how to store the normal direction, or to store the point it hits (all of which the mouse pick can do).
« Last Edit: April 02, 2017, 10:56:09 PM by westingtyler »

andycoke

  • Playmaker Newbie
  • *
  • Posts: 13
    • View Profile
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #9 on: November 16, 2017, 09:09:25 AM »
hi westingtyler, did you manage to get to the bottom of this? I'm stuck with the same issue.

djaydino

  • Administrator
  • Hero Member
  • *****
  • Posts: 3523
    • View Profile
    • jinxtergames
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #10 on: November 18, 2017, 08:33:31 AM »
Hi,
Screen pick and screen to world point are in the standard action list, have you tried them?

andycoke

  • Playmaker Newbie
  • *
  • Posts: 13
    • View Profile
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #11 on: November 20, 2017, 04:59:51 AM »
Hi djaydino. Yes, I've looked at them both, they both say they use the Main Camera. I need to take the pick from an Oculus camera.

Originally I used uGuiVrGazeButton from the Ecosystem. It works fine if using without a headset. As soon as I put a headset on, the pick is below centre. It's so close but no good as it is. I posted about it here -
http://hutonggames.com/playmakerforum/index.php?topic=16074.0
« Last Edit: November 20, 2017, 09:45:27 AM by andycoke »

djaydino

  • Administrator
  • Hero Member
  • *****
  • Posts: 3523
    • View Profile
    • jinxtergames
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #12 on: November 21, 2017, 09:04:41 AM »
Hi,
I do not own an Oculus (yet) so i can't really test and find a solution.
I did try the uGuiVrGazeButton.
Is the gaze point not in position, if so maybe you can adjust the position from it to match.

Also maybe you could tune the position in the 'Gaze' FSM.
on the states "Gaze Enter/Exit Watcher"
Between 'Rect Transform World To Screen Point' and 'Event System Current Ray Cast All'
Do a "vector 3 Add" if you need to deduct you can use the negative sign (-1 for example)

Again i can't test so i am not sure if this would work :)

westingtyler

  • Full Member
  • ***
  • Posts: 192
    • View Profile
    • My Oculus Rift + Touch VR Game Using Playmaker
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #13 on: November 29, 2017, 06:40:15 AM »
no, I was never able to get to the bottom of this. I just tolerated the problem until I got the Oculus Touch controllers, at which point I abandoned keyboard and mouse support out of necessity. I would like to solve it. I haven't checked if the problem still exists in the 2017.2 version of Unity or whatever.

Amin

  • Playmaker Newbie
  • *
  • Posts: 12
    • View Profile
Re: Mouse Pick is WRONG in "Virtual Reality Supported" Mode
« Reply #14 on: May 27, 2018, 10:51:52 AM »
Hi,
 screen position reference is different then mouse position, y is inverted ( bottom/top). I can't remember which one exactly, but there is a catch here somehow :)
bye,
 Jean
Hello Mr.Jean ...
could you please update the uGuiVR sample ?
Most of the friends have problem with it !  :-\
Please fix it sir.
thanks