Okay finally got it working,
I had to uninstall vortex, delete the 2 files Pickysaurus mentioned in appdata\vortex\plugins\
as well as the same 2 files in \roaming\ vortex\ on the drive where vortex is installed
Then reinstall vortex, and activate Skyrim vr in the games section.
After that everything was back, just had to enable all the plugins again and deploy.
Update - Ok so it seems this is a band aide since it seems I have to do this everytime i restart my computer.
