Skyrim Virtual Reality Experience, from the author of Thu'uMic

[DeadlyAzuril]

http://img442.imageshack.us/img442/1383/skyvr.jpg

 

Hi All,

 

I hope a few of you have had a chance to use my last mod, ThuuMic - a mod that allows for voice recognition for shouts and character interation in Skyrim.

 

This was the first step in a much broader concept - A complete virtual reality re-haul of Skyrim.

 

Now I hear you ask, what does this entail?

 

By combining visual, aural and kinectic (touch) stimuli - we can truly immerse ourselves in the world.

 

VISUAL

• Always 1st person Mod, including Horse riding, Killcams and player actions (sitting, crafting, alchemy, enchanting) - IN PROGRESS
  
• 3D Positional/Rotational IR Head Tracking / Via Natural Point Track IR 5 - IN PROGRESS / LIMITED FUNCTIONALITY
  
• 3D Head Mounted Display / Via Sony HMZ-T1 720p 3D HMD - NOT YET IMPLEMENTED
  

 

AURAL

• Thu'uMic voice recognition / Via in-built microphone and ThuuMic software - DONE
  
• Surround Sound noise cancelling headphones / Via Turtle Beach 7.2 Surround Headphones - DONE
  

 

KINETIC

• 3D Positional/Rotational hand tracking and gesture recognition / Via Razer Hydra Motion Controller - NOT YET IMPLEMENTED
  

*note in this list I also list the particular hardware that I will be using to accomplish these tasks, Though know the work done will also apply to cheaper/free alternatives such as freetrack, wiimotes, PS move controllers - as I wish to make my findings available for use for the community. also the modular nature of the project will allow users to pick and chose which modules they would like to install. we cant all afford a 3D heads up display (myself included - im doing some saving haha) so cheap accessibility is important (ala Thu'uMic)*

 

With this hardware and the correct implementation via Modding, I hope to allow for a full VR Skyrim experience, heres a more detailed breakdown of each module.

 

Always 1st person mod

WHAT IS IT?

This mod will kill the third person camera, replacing killcams, horse riding, crafting, alchemy and effectively all third person animations with 1st person counterparts. Nothing is less immersive than being pulled out of the head of your character.

 

This will also add the full character body (with shadows) into first person, and finally - I wish at the end to model in different irst person masks/helms/and hoods that you can 'see through' this will ofcourse be optional. but i think it would be amazing.

 

HOW

A while back I proposed a few different ways in which an always 1st person mod could be achieved (link), turns out one such way - messing with the camera inside the character .nif files were the first step in the right direction. I am working closely with 3AMt on fixing his 1st person implementaion. You can follow progress on his implementation here!

 

3D Positional/Rotational IR Head Tracking

WHAT IS IT

Using an infra-red camera and infra-red LED's (or reflectors) we can extract true positional and rotational data from the users head. With not only rotational, but POSITIONAL data - we can now have our characters get on thier toes to look over an obstacle, peek around corners, or follow a dragon as it flies overhead by just tilting your head. The implications of this is immediatly apparent in their usefulness in terms of immersion.

 

view

to see what I mean.

 

HOW

This is a tricky one, with more than a few steps to get working *fully*. The main issue we face here is that there doesnt seem to be any way to influence camera direction via the CK. thus, a few things are required before we can say we have true 3d head tracking. they are as follows.

 

• unlocking camera look and aim
  

The camera in skyrim is 'locked' in that your body will always rotate to the direction you are looking. for this to work correctly - we want the head to rotate independantly, so the when the body is facing north, we can still look east/west without rotating the body. ofcourse mouse/controller rotation will in this case specifically control aim, but will rotate the head if the angle strays more than say 100 degrees either way. I am currently looking into different implementations of this. but the way I feel it would be best incorperated is by breaking the third person camera, and locking it to the *head*, allowing for free rotations outside the the main camera.

 

• unlocking camera position
  

Ok so this is where it starts to get tricky, we can not only read ROTATIONS from head tracking, but POSITIONAL DATA also. what does this mean? Try - leaning around corners,peaking above the table your sneaking past to steal something on top, getting close to look in at something interesting. To implement this, I have to refer back to the camera 'hook' so we can take in the positional camera data, but then add an offset based on our positional distance from our virtual head tracking 'centre' or world coordinate zero.

 

So for this, we need to find this data, so we can update it ourselves in memory.. not easy. :/ this will no doubt require SKSE or Script Dragon to implement.

 

An interesting side note is that skyrim uses Havok Animation which has full support for head tracking and IK setups that allow for true leaning and input data,and also has an available SDK for download. but whether or not this is at all useful at runtime is beyond me at the moment. lots of work to be done.

 

NOTE: TrackIR *does* have mouse emulation for pitch/yaw - which i have used successfully in skyrim. but it is no means a true solution and in know way shows off the true power of true rot/pos head tracking.

 

3D Head Mounted Display

WHAT IS IT

With head-tracking, the next logical step is VR goggles, right? ....RIGHT? The Sony HMZ-T1 is a head-mounted display unit with 2 720p OLED screens for true stereoscopic (3D) vision, which will be enabled through Nvidia 3D vision drivers. This should work out of the box, though I hear stereo in skyrim has one or two issues which might need addressing, but this one is pretty self explanitory and doesnt require much, if any modding to implement.

 

HOW

save $$$, these puppies retail for $899 AUD

 

Thu'uMic Voice Recognition

ThuuMic is a mod developed by myself and psychoHampster, and allows the use of a microphone for enabling shouts and interacting with NPCs (call horse, companion orders)

 

There is still one or two things I wish to accomplish with thuumic, one is find a way to 'teach' it your own voice per-word for better recognition, and the second is adding new companion functionality (wait for me, change between melee/ranged, setup ambush, distract, hold back, hide)

 

a lot of this functionality is now available to mess with via the CK, I will eventually get to these.

 

Surround Sound

Incredibly important and probably the easiest to implement, combined with head tracking and VR, true 7.2 audio from headphones will accurately move with your head-rotation, resulting in fantastic immersion.

 

3D Positional/Rotational hand tracking and gesture recognition

This is the big one. and will take the most work, easily. Using dual motion-sensing controllers (Razer Hydra), imagine having your hands acurately tracked - an individual free-aim for both arms. This has a lot of implementations and is very exciting in its prospects. Now you may have seen a video online recently where somone is using a hydra in skyrim. and lets face it - its a terrible implementation, the same mistake all of these motion controlled games make, which is deadzone turning based on controller rotation or aim.

 

This is a broken way of doing it and destroys the immersion. even in ps3 move titles such as killzone, or wii first person shooters. they are all guilty of this practice. Im going to talk you through what i would consider a much better alternative, with true simulated immersion for the player.

 

Individual hand aiming and orientation, combined with a overhauled combat system for gesture recognition.

 

Sword/Dagger/Axe/Mace

swing your equipped hand to swing in that direction, powerful strike for a power hit. stab for a stab move. *perhaps integrate true blocking?*

 

Shield

Raise arm to raise shield, bash with arm raised to do shield bash

 

Bow

Hold right trigger and pull back hand to gain tension on bow. aim with left hand, mimmicking holding the how. release trigger to fire.

 

Magic

A completely rehauled magic system, combining many spells to streamline magic use.

 

Instead fire rune, fire breath, fireball, fire shield (i dont play a mage, these are just examples) you just have "fire"

 

With "fire" equipped..

Hold trigger then release forward to shoot fire ball // gesture used for 'shooting projectile magic'

hold out hand then hold trigger to shoot flames from your hands // gesture used for 'constant magic stream' such as healing hands

hold up hand and hold trigger for flame shield // gesture for magic cast on yourself

point hand at floor and pull trigger to cast fire runte // gesture for rune casting

 

with the method, magic can be VASTLY streamlined.

 

Individual aim

One important thing that is wanted to achieve is to feed the positional/rotational data directly into each hand obect, so you could in effect fire in two different directions at the same time, this becomes even more usefull when we think about object manipulation and sneaking

 

Object manipulation/ Sneaking

Rather than have a 'use' button to pickup objects. I wish to have a collision box attached to the unarmed hand, so pulling the trigger in unarmed mode will query which object you are reaching for (say your reaching for an apple) and then instantly pickup and equip the object to that hand, effectively letting you 'pick up' objects.

 

Part of this is then enabling clutter to be 'equipped' - which is possible inside the sdk. I think that the rotational/positional 'hook' will be the hardest thing to get together.

actually calling on this collision event and equipping clutter i dont see as too big a hurdle.

 

This would also apply to pickpocketing, where in sneak mode reaching to an NPCs pocket will enable the pick pocket menu, and lock picking will take rotational data, so you feel like your 'picking' the lock.

 

* Minimal Implementation : I am certain that simply enabling *some* of these things can be done simply through gesture recognition sending macros to skyrim. so although we can probably get this working rather easily, it would again not be a true representation of the what the technology is capable of. I will be updating as I implement more **

 

- - - - -

 

So, theres my idea. and I am indeed going ahead with it. Though If anyone with script extender experience can perhaps shed some light on some of the harder elements (camera hooks, feeding data from outside skyrim into the game) please let me know. I think the community as a whole will benefit from the ability to utilise these technologies, So my intention is that this post can become a hub for discussion into possible implementations of these ideas.

 

Cheers for reading, thoughts?

 

Az

Edited February 14, 2012 by DeadlyAzuril

[TheRagMan]

thoughts? LOVE, a great step for immersion and accesibilty.

[Rennn]

Yay. I already have TrackIR :)

And here I thought it was totally useless.

Now I'm actually glad I own it.

[Khet]

Holy crap is this real? I gotta figure out how to get my hands on the equipment required to do this. Experiencing Skyrim like this would be... incredible. This is definitely something I'll be keeping a very close eye on!

[Jakisthe]

Most abitious mod ever?

 

Yep. :thumbsup:

 

Requires users to have $5250 worth of rarely preowned equipment each to play?

 

Yep. :ohdear:

Edited February 14, 2012 by Jakisthe

[DeadlyAzuril]

@Khet it is very real my friend, as I make progress with different modules i will post videos following the process.

 

At the moment, mouse emulation works with skyrim, but as I use a 360 controller (and soon will be using a Razer hydra) this doesnt work, unless I can determine a way to take both controller and mouse input simulatneously

[DeadlyAzuril]

Ok, update. as it is my birthday, my girlfriend kindly let me down easily saying the hydra would takes two weeks. apparently shes a gloriously deceptive girl and now i have the hydra in my hands. so expect progress on that to move along much sooner!

 

*so awesome*

[CylonSurfer]

I'm very interested in this mod. I'll be keeping a close eye on this thread.

 

I own a HMZ-T1 & have a 3 point headtracking system setup for it using FreeTrack. As you mention the mouse emulation solution offered by both TrackIR and FreeTrack is great, but the full implimentation of head tracking in Skyrim would make it the first affordable VR gaming experience in homes.

 

At the moment, mouse emulation works with skyrim, but as I use a 360 controller (and soon will be using a Razer hydra) this doesnt work, unless I can determine a way to take both controller and mouse input simulatneously

 

I have a 360 pad hooked up to play Skyrim, while using FreeTrack for headtracking I enable both K&M & the 360 pad ingame by using a program called X-Padder. This allows me to map key strokes and mouse movement to 360 pads buttons and anolog sticks. The only downside I have experienced thus far is that walk/slow walk has to be toggled due to the limitations of the key stroke movement controls. Oh, vibration is also off.

 

Off course a way to enable both inputs without the need for a 3rd party program would be preferable.

 

If you plan on getting a HMZ-T1 then I'd recommend you hurry (finances permiting). There are rumours abound that the HMZ-T1 will have a very short production life, I have even read reports of people contacting Sony to be told that this product is no longer available in their region. I'm sure we'll see a HMZ-T2 if that is true.

 

If you make any progress on the Head Tracking front, feel free to hit me up for some testing, I'd be happy to help out. From what I have read on the 3D Vision forums, the vast majority of the 3D issues have now been fixed, if not all of them.

 

Good luck!

[DeadlyAzuril]

Awesome, thanks for the heads up mate.

 

I'll definately hit you up for testing when ive made some progress. The display isnt available in aus yet, so as long as I can secure a pre-order in the next few weeks i dont see it as a problem, but hey who knows.. *fingers crossed*

 

As i said i also have the hydra now so realistically i have everything i need to get into full swing. if only i didnt have to work today! :)

 

ill keep you posted

[Ikedude]

I've been looking into doing 1:1 tracking of the hands via Wiimote or Move. The libraries out there for the Wiimotes are kind of outdated and few, if any, make use of the motion plus addon (necessary for this application).

 

This framework for the Move seems to be a little more full featured in that it tracks rotational and positional data and updates it for you using the data from the Eye and the wand.

 

Move Framework

 

Is there a framework like that for gathering data from the Razer? Or would you just be using DirectInput?

 

Getting that data into Skyrim is a whole different matter though... I've been trying to figure out how you could inject that data but I haven't gotten very far. Looking at the Smart Souls source:

 

Smart Souls

 

It looks like he is overwriting functions that get called when a soul is captured. I don't know if this would work for this application... (BTW, if anyone knows how to find out where these functions are stored in memory, I would love to know)

 

The way I'm assuming animation works is all of the animations are loaded into memory (idle, running, attack, etc.). For each frame the appropriate animation is updated or a new one is started. When the animation is updated it pulls the next frame from memory, this would be where you could hook in and load your data instead. The problem I see is that the player isn't the only actor that calls this UpdateAnimation (or whatever its called) function. If there is a way to find out what actor is calling this function it might be possible to only affect the player character.

 

Does anyone know if the collision detection in game works for non-canned animations? Say if you changed the animation to look completely different, would it still have the same hitbox or does it actually depend on the location of the sword/axe/whatever? If it is just detecting the weapon mesh hitting another mesh and not some pre-defined hit area then that makes it a lot easier. Also, it would be cool if you could scale damage based on how fast you swung.

 

This is pretty ambitious but I agree it could make the game much more interesting. Combat would go from swinging randomly to actually trying to block hits and maybe parrying. Combine that with locational damage (I think Duke Patrick was saying he was going to try and implement that) and combat is completely different.

 

I'll definitely be following this thread.