Potential new utility for merges and large load orders. Anyone interested?

[primem0ver]

LATEST NEWS SHORTCUT... latest news is HERE

 

IMPORTANT UPDATE: This utility is now capable of fully converting VOICED NPC mods to an ESL. Check out the demo here. So far I have tested it on 3 voiced followers. See the news link above for details.

 

 

ORIGINAL POST:

 

The short version: I am wondering if people would be interested in a utility I am calling MSM or Mod Set Manager. It allows you to organize large collections of mods into sets using tags and a spreadsheet. Those tags are then used by the utility to automate many of the painstaking and laborious tasks involved with developing merges in order to minimize a load order (particularly when they are big).

The long version:
I am a "mod-ohoolic" who has used myriads of mods in combination throughout the years. I have never been one to wade through the massive painstaking processes that are involved with merging large mods and have developed my own suite of utilities to make the job easier and to make the task of getting around Skyrim's and other utilities limitations bearable.. In the past I have used collections of somewhere between 500-600 mods using merges that I would track using excel and exported to a csv spreadsheet (because excel was such a pain to deal with in early days). Using the spreadsheet I would "tag" my mods to be organized into sets which I could quickly activate so that only those mods would be part of the load order. I would then merge them and use another utility I made to find missing resources (Merge Plugins doesn't always get everything needed) before restoring my load order. The separate utilities would deal with things like only activating certain mods and their masters; copying resources to the new merge to be sure they weren't missing, deactivating mods that have been merged to reduce the number of esp's that end up in the "data" folder, etc. (Anyone who had more than 500 esp's in their data folder probably ran in tho the bug where the player character would be headless... and other strange graphic oddities). Even so, there were many difficulties and annoyances with my old utilities.

This is why with my coming back to Skyrim and deciding to switch to the Special Edition I have built a single utility from the ground up that accomplishes everything that the old utilities did and much more. I am building my first load order with everything I need to quickly figure out what mods could work potentially merged and which ones could not. After that, (assuming everything works), I should be able to build all my merges within an hour or so rather than a day or more. While looking through various mods and doing things to help my my tool more useful, I have noticed that there a lot of people like me out there who like lots of mods and I felt that if people wanted, I built this for myself but I could share. So...

Without making any promises...

I may decide to share what I have built with the community. Please keep in mind that my main goal is to actually play Skyrim again instead of spending a thousand more hours programming. Depending on people's interest in this project, I may share it on GitHub so that people can contribute and carry on if I decide to stop working on it.

Basic Requirements:

• Mod Organizer 2 (A Vortex version may be negotiable but NMM is s definite no. I can't accomplish the goals of this software if everything is actually in the Data folder).
• Merge Plugins (I looked into zMerge and left a message with the author but so far no info on whether zMerge keeps data on the merges it makes. Plus... some people are reporting major issues that made me shy away from this tool for the time being).
• For now: some way of being able to create and edit Excel documents. This can be done with Google Docs.

Features (A little taste of what it can do right now):

• Full Excel File support (currently requires a very specific layout but I may make the layout dynamic in the future).
• Allows you to organize the order of mods in the excel file how you like (independent of load order).
• Use category dividers in Excel to specify a mod category. All Mods underneath that category will be added to the category.
• Categories are loaded and easily viewed in MSM.
• Easily switch between the full mod set, tagged sets, and what I am calling the playset (all mods which are active for a Skyrim game/playthrough).
• Integrates Merge Plugin profiles and manipulate the merges outside of Merge Plugins.
• Does not require starting through Mod Organizer. In fact, it can Start Mod Organizer for you.
• Automatically shuts down Mod Organizer when activating a mod set and restarts it when finished.
• Allows you to easily control how each mod is loaded into the game whether it is merged, an ESL, or a fully loaded mod (merges must still be built using Merge Plugins).
• Give titles/reminders to mods to let you know what they are
• Easily locate mods you forgot were ESLs and mods you want to make into ESLs if possible (by flagging them to be loaded as an ESL).
• Compact an ESP and convert it to an ESL at the click of a button.
• Batch (compact if needed and) convert ESPs to ESLs all at once or as groups to quickly prepare a collection of new mods for a new game.
• Gives space for up to five comments (in separate Excel columns) for each mod.
• Gives detailed feedback on the status of merges and mods within the merge.
• View configuration errors (inconsistencies) which exist for any given plugin mod.

Included Tools for Building Mod Sets/Merges Relatively Quickly:

• Dependency Viewer: View all mods that require a specific mod as a master/dependency (inverse of viewing all mods that are the master of a specific mod).
• BSA Viewer: View the structure of a mod's BSA to help decide if it is a good candidate for merging.
• Mod Set Activation which includes a detailed, color coded preview of all masters, set masters, and requirements. Mods are loaded in the order they are needed in order to build a merge.
• Merge Resource Builder Makes sure non-BSA resources are also included in a merge build. (I have had lots of trouble with missing textures and other resources in the past).

Some Screen Shots
(Please understand that these are preview/preliminary and include the number based tags that were required before in my Oldrim utilities/setup.. Tags can now include any character that can be used in a file/folder name.)

Sample Spreadsheet:


Elsewhere...


All Categories:


The Location Category:


Log/Status/Error feedback tab:


Master Check Tab:


Mod Error window:


Mod Menu Options:


Mod Set Activation Tab:

Edited July 18, 2021 by primem0ver

[HadToRegister]

I used to use Zedit and MatorSmash but since my mod list went over 254 Plugins, I can't use IT, or ZMerge, I've heard that the author doesn't want to update the program to accommodate more than 254 plugins.

Which is why I would like to see your program/idea come to fruition.

Which was WHY I mentioned those programs like you did, to let you know that those programs had limits.

[primem0ver]

Mktavish.... without having seen their other posts... I found the post informative. I have heard the feedback about zEdit mentioned by others but didn't realize that was the choice of the mod Author. (Though 254 total plugins is a bit on the low side for a tool like this lol).

 

EDIT:

Regardless, I don't think I have merged that many plugins together before. Even if that is the limit by zEdit, unless someone wants to merge in excess of 254 mods together, this tool can help get around that limit.

Edited May 17, 2020 by primem0ver

[HadToRegister]

Mktavish.... without having seen their other posts... I found the post informative. I have heard the feedback about zEdit mentioned by others but didn't realize that was the choice of the mod Author. (Though 254 plugins is a bit on the low side for a tool like this lol).

 

 

Well technically it's 255, but it starts from 0.

 

I love ZEdit, and MatorSmash, but according to other users he's not going to make it support past the 254/255 limit, which is a bummer, because those were my 'go to" tools, I especially loved the Zpatches, where you could make 'tweaks" like you can in WryeBash with the "Tweaks" feature.

But i stopped working once I passed the limit, so I'm looking forward to your idea.

That's why I mentioned the Z programs in the first place

[Wolfpack49]

Sounds interesting. You may want to share on the Steam community or on /r/Skyrim mods. You may get more feedback there.

[anjenthedog]

No, because I don't use MO2, and therefore, since it is a requirement, I would not be able to use it.

 

However, it's a great idea and kudos to you for embarking on this effort. Shame you're only supporting MO2, but I completely understand.

[bloodstone2008]

I understand the limit Zedit is using. If it can load the limit, why try and exceed what is allowable bye the engine. It also makes sence that you would want to pack up your Mods, in a logical fashion to find errors and bugs in mods.

[primem0ver]

Thanks for the feedback so far...

No, because I don't use MO2, and therefore, since it is a requirement, I would not be able to use it.

However, it's a great idea and kudos to you for embarking on this effort. Shame you're only supporting MO2, but I completely understand.

Do you use Vortex? If so, I want to hear from people like you because as I mentioned in my OP, I may visit the idea of implementing a Vortex version if I have the time and enough people want it. I can't do NMM though because it doesn't keep mods in separate locations which is a requirement for the operations that are performed by this utility. I need to easily and quickly figure out what which files belong to which mod. If mods didn't have any loose files, I might be able to create a version for NMM... but that can't be guaranteed. If enough feedback and requests are made, perhaps I will consider a NMM version with the features that require the separate mods disabled.

I understand the limit Zedit is using. If it can load the limit, why try and exceed what is allowable bye the engine. It also makes sence that you would want to pack up your Mods, in a logical fashion to find errors and bugs in mods.

I haven't looked into zMerge well enough to know exactly what this means... but if I understand it correctly, it only means you can't merge more than 254 mods together and that if there are more than 254 mods are in the load order they won't appear for merging. I have three important things to say about this:

• Dealing with merging mods when there are more that 254 mods in your "master" load order was one of the big reasons I created this program as well as their predecessors. I was always running into problems with large load orders The whole point of this software is to keep track of your entire load order while at the same time allowing you to activate only part of your load order so that you can easily merge them together. EXAMPLE: You have 1000 mods in your "master" load order but you want to merge 120 of them together. Solution: You create a "tag" for those 120 mods and then apply the tag to those mods. This creates a "set" of mods that you can then activate in order to merge. When you activate a set, only the set's mods and their masters are kept enabled, ALL OTHER MODS AND THEIR MOD FOLDERS ARE DISABLED temporarily. That leaves only those 120 mods + their masters in the load order (plus any mod with the master flag set to true).
• If a person REALLY wanted to merge more than 250 mods, I suppose this program wouldn't help with that... but I am not sure why anyone would want to merge that many mods together, This utility allows you to merge a good number of mods together. There is no set maximum as long as the set you want to activate + masters is less than 254 mods.
• Currently I am not supporting zMerge. The reason for this is because as far as I know, zMerge doesn't keep track of the mods it merges. If this is wrong, hopefully the author will let me know (as I left him a message in his issue reports on GitHub). The other reason is because some people are having HUGE problems with zMerge deleting all their mods when it crashes. I will NOT go down that road. If these two issues are fixed, I will definitely support zMerge with this software in the future.

Edited May 25, 2020 by primem0ver

[FullTimePatriot]

I'm very glad you're looking at this project with Mod Organizer 2 in mind. It has been a godsend for me. Between that and loot I've managed to get my game modded with only about 20 hours of troubleshooting compared to how painstaking the process used to be, for someone of my limited intelligence at least.

[anjenthedog]

primem0ver,

 

Yes, I use Vortex. I'm tracking this thread (and will track any mod you *do produce), because the idea is cool, so I'll keep a watch.

 

Thanks for considering us. I understand if that doesn't gel for whatever reason.

 

best wishes