Sharing my opinion on mod management system

[marius851000]

A piece of text I wrote today about sharing some things I like with certain mods manager, and might be of interest in the NexusMods.App team. It’s stuff I’m thinking about for a few years (due to my interest in package management), so I wrote it down, as you might be interested by it.

 

=== My opinion on an "ideal" mod management system ===

Hello. I’m someone who’s interested in modding (but on PC, it’s mostly limited to Morrowind).

I’ve read the design document of the new proposed NexusMods.App, and given that I’ve spend a lot of time developping opinion on what should be, in my point of view, and ideal mod management system, I’ll share it here.

As far as system I use, I just use Portmod for Morrowind, a package manager based around the concept of portage, the package manager of the Gentoo Linux distribution. I’m also using the NixOS Linux distribution, which is an immutable and declarative operating system with configuration files in a functional programming language (called Nix).

I’ll do short presentation on how they work, as I’ll refer to some of their concept and compare with them,

==== Portmod ====

Portmod is a mod manager, that share a lot of similarity with traditionnal package manager (which is kinda what Linux users use most of the time to install software).

It uses a curated mod repository metadata (each mods is a Python class, evaluated in a sandbox). These classes contain info on how to fetch the mod (just basic HTTP/S fetching, or a file name the user should download themselves). It then also contain a list of things to install, that will be installed in a target folder (one folder per mod). It may copy ESP and other data files, as configured in this class. The class may also contain some information on what changes should be applied to the configuration file.

One nice things about those dyanamic declaration it automatically apply (or not) compatibility patches when needed. Mod installation can also be configured with USE flags (which will impact the result folder). Portmod is able to detect when installing a new mod whether it should re-install another mod that need to have a compatibility patch enabled (or when a USE flag change), but will in that case replace the previous mod. It also support automatically updating mods (when metadatas are updated).

Portmod support multiple games, but I’ll focus on Morrowind via OpenMW. OpenMW have native support for VFS, so enabling a mod amount most of the time to patching the openmw.cfg file to add a "data" entry and then one or two for esp/bsa/esp but for grass (and sometimes a few others for enabling bump map or things like that).

==== Nix/NixOS ====

Nix is a package manager, used by the NixOS distribution (note that all I say also apply to Guix and it’s distro GuixSD, which is derived from Nix, but uses a different repo and use the Guile (a Scheme variant) language instead of the Nix language).

The way NixOS work is that it declare packages as set of build instruction (defined in a Nix file), that’ll be used to build the package. It can depend on other packages. Then, all the build inputs are hashed, the output folder determined as something like /nix/<hash>-<name>/, and the build is executed in a sandbox.

This mean that any change in any input (or the package) itself will result in a different hash, and so, an already installed packaged will be replaced (but may be deleted if not other packages/other reference it)

The whole system is build that way, including the boot-up script for NixOS, systemd services, (sometimes, depending on the configuration) configuration for services (thought they may sometimes just be stored in /etc/ as standard file the user can manually edit), the PATH and everything else that’s needed for a standard Linux environment (Nix also work on FreeBSD and MacOS, but just for installing packages).

It have two main interest:

* It can perform rollback with ease. You can select to rollback to a previous (non-deleted) revision. It’ll just change the start-up script, and you have a roll-backed system without needing to override files (notes: it actually need to replace some symlinks in /etc, thought it support storing /etc in tmpfs and starting from a clean one).
* Rollback from running system may be more complicated, but it’ll restart systemd services that need and update the environment variable and co.
* It can have multiple revision/version of a package installed at once. It can be usefull for having a development environment with (for example) old version of packages (or a patched package).

(plus a few others that’s more technical)

==== My suggestion ====

There are two parts I consider important:

* How to install a (single) mod
* How to store the files

===== Install a single mod =====

So, here, I’ll just focus on the thing that I consider are things that are different between usual mod management system and my opinion. So I assume that:

* The mod installation is fully automatic
* The user can enable or disable certain things in the mods via some settings
* There may or may not be some game-specific code that can handle certain game-specific aspect of the installation (like esp file cleaning)
* A mod is installed as a whole (atomicity)
* The mod order coherency is automatically guaranteed (so dependancies are present befored the dependant)
* A mod may be updated if the user enabled/didn’t disable this option, but there should be not automatic update to breaking version (that also mean mod author know what is a breaking update or not, but in case they doesn’t, it could just be assumed every update is breaking and no automatic update will ever happen)

In particular, I do not assume that every mods are in a single, coherent repository (thought I still consider having such a repository is important. It is expected that mods that are either common, hard to install coherently with other mods, or lack enought metadata for proper automatic installation will be here, but it should work well with other mods)

====== Build-time access ======
There’s also the question of what a mod should be allowed to access during installation.
* Should it only access mods it directly depends?
* Mods earlier in the mod order (which will prevent multi-threaded installation, and need to reinstall all later mods when a mod is added earlier in the mod order)?
* All currently installed mods (impossible to decide when to reinstall it)?

In my opinion, a mod should be in one those three category:
* Do not depend on non-dependancies (thought there may be optional dependancies, which will make a mod a dependancy of it will be installed even if not a dependancy)
* Mods that need to be run on each mods, but not the final set (for example, something that patch .esp file individually). Those will be added at the end of the installation instruction of a mod.
* Mods that need all other mods but those in this category. For example, ensure coherency between each mods. They are build at the very end, and have access to the entire file system (but their order are still important between them).

====== Automatic option enabling/disabling ======
A mod should be able to enable or disable certain of it’s feature (like HD texture or compatiblity patch) based on which other mods are installed or game-wide user option (that may optionally be overriden on a per-mod basis by the user).

Something portmod does is that a mod may have multiple download, for exemple for downloading compatibility patch if needed. In my opinion, this kind of things should still possible (download further data based on the enabled options), but said optional download may or may not be split into a dependant mod (I think either of the choices are goods).

This allow to install a modlist just by specifying the source of the mods. Mods will then have enought metadata (either embedded or via the coherent repository) to have a load order selected.

===== Storage and patching of games =====
So... I read the document about the proposed "immutable modlist". And I think it’s a good idea, thought the "install a single mod" part is planned around the idea that each mod are first extracted, it may also work by not extracting files at all and just manipulating virtual files most of the times, thought some files may be needed to be extracted anyway for certain patcher.

(note that both of these assume a transparent VFS is used (I know this is possible on Linux with FUSE, but no idea for Mac or Windows), or at least a way to patch files together, which is what your immutable mod list permit)

Or it may be a VFS system built-in into the game, like for OpenMW, which is setup by writing to some file.

====== A per-mod folder proposal à la Nix ======
Here, each mods are installed in a different folder, including for updates. This is like your immutable modlist, but with one folder per mod, folder which is generated by the build instruction (which may also store all inputs so it can be reproduced if needed). Note that also changing use flags will result in a different folder.

====== A per-file file ======
This is similar to the modlist approach, but instead of keeping the file virtual, they are all written to a folder with a path like "<something>/files/<hash>.bin", and then the modlist will decided which of those to use (without ever writing any file, given it’s patching using a VFS).

After the building of a mod is completed, all files are hashed and moved here if not present, and the hierchy with map to the hash is recorded somewhere to then be used by a VFS.

====== Final thought about that ======
The idea you propose for patching file is very good, thought I still consider it important to be able to isolate building the mod and adding it to the game file system.

Most installation operation can be done without ever writing a file for it, but some amount of writing will be needed anyway for patching files, and then those files will need to be stored somewhere. (or maybe just rebuild the mod everytime on of their patched files is needed?)

==== All-encompassing final thought ====
I hope this document might be usefull to share some perspective on ways to manage mods. What I tried to do here is to make installation of mods as easy as possible, while still allowing complex situation, with a particular focus on installing many mods (like the graphics-overhaul, about 100 mods installable in a single command with portmod modulo manually downloading 50 mods on NexusMods (the other half either being redistributible so mirrored or hosted elsewhere)).