Using Mod Organizer 2 on Linux the Right Way

[Mummelpuffin]

Just managed to do this myself, and I'm very happy to understand how to handle this in the least janky way possible, finally. No install scripts needed, easily manage multiple games. Leaving these instructions here because this is so poorly documented on the internet generally.

Step 1: Grab MO2 from it's GitHub page, specifically the 7z archive. It'll be named something like Mod.Organizer.(version number).7z.

Step 2: Extract MO2 into a folder somewhere. Not into the folder of a Steam game, in it's own folder.

Step 3: In Steam, add your newly extracted MO2 as a non-steam game (Games, add Non-Steam Game to my Library, Browse, find ModOrganizer.exe...) DO NOT LAUNCH MOD ORGANIZER YET.

The following steps must be repeated for each game you wish to use with Mod Organizer.

Step 4:
Need some background info here. Proton (and WINE in general) installs a WINE / Proton "prefix" for every Windows game you install. A WINE "prefix" is (roughly) it's own self-contained virtual Windows installation. While the files of your game are sitting in their usual folder in Steam, the game's WINE prefix files remain in their own folder. These will include any files the game creates after launching (config files, save files, etc.) if they're meant to be stored in My Games or something rather than the game's own install folder.  These prefix folders are stored in /steam/steamapps/compatdata. Each game's folder is named after it's Steam app ID (usually? There are weird exceptions, but don't worry, that's why we're finding the folder name through ProtonTricks.)

By running Mod Organizer through Proton, Steam is going to make a prefix for Mod Organizer, with it's own compdata folder. The problem here is that when Mod Organizer launches a Steam game, it'll be using Mod Organizer's compatdata folder rather than your game's... which can cause issues. Steps 4 and 5 fix this by forcing Mod Organizer to use the game's own compdata folder instead.

You need to know what your game's compatdata folder IS, and this is where I suggest installing ProtonTricks. It's available as a flatpak, so pick it up through your distribution's "app store" or whatever. It functions as a wrapper for WINETricks (don't worry, you don't need to know what that means). When launching ProtonTricks, you are presented with a list of Steam apps. The number next to the app's name is the title of the game's compatdata folder. Got it? Good. 

Step 5: Select your Mod Organizer entry in ProtonTricks. We need a dependency from Windows land for Mod Organizer to actually run through Proton. Go through the following menu options:
Select The Default Wine Prefix > Install A Windows DLL or Component
Scroll all the way down until you find vcrun2022. That's what you currently need for Mod Organizer to launch (or, at least, it's one of the Visual C++ libraries contained within. If this is the future you might need to get creative.) Check the box, hit OK, wait for the installer to pop up and do it's thing. You can close ProtonTricks.

Step 6:
Enter Steam's properties menu for your new Mod Organizer entry. I suggest naming the entry something like Mod Organizer [game title here] because it'll be game specific. Enter the following launch command: 

STEAM_COMPAT_DATA_PATH="/home/isabelleboucher/.local/share/Steam/steamapps/compatdata/22380" %command%

Obviously the specific data path needs to be adjusted for your Steam installation and for the compatdata folder you just found through ProtonTricks. 22380 is the compdata folder for New Vegas, if you're curious. 

In the compatibility section of the properties menu, force Mod Organizer to use Proton Experimental (once again, if this is the future, perhaps there is some "older" version of Proton you need to select).

Step 6:
Open Mod Organizer, finally, and create a GLOBAL instance for your game. This will put all of the game's modding files in the game's own compatdata folder. If you create a portable instance, the files will go in the folder you installed Mod Organizer to. In theory you can only do that once, but since we're running Mod Organizer through a different prefix for every game, you might be able to trick it into overwriting your one portable instance. Not sure. Just stick to global instances.

Step 7:
You SHOULD now be able to launch your game through Mod Organizer. If you can't you haven't read carefully or you weren't able to launch the game directly through Steam anyways. Meanwhile, you can still launch the vanilla game if you wish. 

The specific game you are playing may want a specific version of Proton. If you set it in that game's Steam properties, it'll still run with that version when running via Mod Organizer.

Your game will also probably have it's own modding tools and special engine extensions which require their own Windows component dependencies. These can be installed through the game's own ProtonTricks menu. This is where my knowledge ends and you'll need to work out what you need. But in the case of New Vegas, the following are good to have:
- xact
- xact_x64
- d3dx9_43
- d3dx9
- vcrun2022

You will, of course, need to install things like script extenders as you would on Windows. Chuck them in the game's folder, point Mod Organizer to 'em and go.
If there are utilities you need to run which should be launched manually rather than through Mod Organizer, launch them through ProtonTricks with the prefix of the game the tool is for and it should work. You may need to temporarily change the version of Proton selected for the game's prefix for that.

Edited April 2 by Mummelpuffin
Misspelled compatdata

[malilie]

So far, trying to follow this guide I am unable to get steam to create an entry for Mod Organizer in the compatdata folder.  Adding it in steam is not doing so, attempting to run it once in steam does not create it.  There is not an entry for it in protontricks either, this is on Mint 22

Edit: Fixed this was my mistake, I did not force it to use proton experimental.  Doing that in steam caused it to create a compatdata folder.

[MrBreadloafNexus]

Couple notes as it didn't work for me the first time around.

 

Not sure if it's the same for everyone, but make sure the mod organizer folder and the game's proton prefix (folder in compatdata) are on the same drive.

Instead of installing vcrun2022 before adding launch options, I set mo2 to run with proton, set launch options (no quotes for file path), launched it for the first time then closed it, and installed vcrun2022 into skyrim's prefix with protontricks, after this it launches flawlessly.

I was also able to create a portable instance just fine by specifying a different file path.

 

Note that I'm using EndeavourOS with steam from the arch repository, with proton experimental. This won't be the same for everyone. Hope this helps someone having the same problems as me