Potential new utility for merges and large load orders. Anyone interested?

[primem0ver]

  On 10/17/2021 at 10:37 PM, VulcanTourist said:

It's the intention that counts! I can wait... I'll spend the time butting my head against the broken Windows UWP system - did you know that Nvidia Control Panel is now a UWP app? - and the myriad of real-world stuff that doesn't work.

 

As an fyi, the paths are configured using an ini file in the main executable directory. So if you know your way around one of those it shouldn't be hard.

 

Here is an example of what it looks like:

[Locations]
ModOrganizer=D:\Gaming\ModOrganizer
MergePlugins=D:\Gaming\Skyrim SE Tools\Merge Plugins
zMerge=D:\Gaming\Skyrim SE Tools\zEdit_v0.6.6.1_-_Portable_x64
ModDownloadFolder=D:\Gaming\Mod Collection\skyrimse
MergePluginsProfiles=D:\Gaming\Skyrim SE Tools\Merge Plugins\profiles
BSAViewer=D:\Gaming\Skyrim SE Tools\BSA Extraxtor v0.10-974-0-10\bae.exe
HexEditor=C:\Program Files (x86)\Developing\IDE\UltraEdit Studio 16\UEStudio.exe
SSEEdit=D:\Gaming\Skyrim SE Tools\SSEEdit_4.0.3h
TESVEdit=D:\Gaming\Skyrim TOOLS\TES5Edit_3_1-25859-3-1

[UI]
PanelDelta=182
RememberSizeAndPosition=false
PreferredWidth=1364
PreferredHeight=810
PreferMaximized=false
UIForceBothCaseCategoryNames=true
RowFormat=Data
SplashProgressColor=FFFF0000
CensoredRun=false
TriggeredUpdateInterval=2000

[Options]
LooseFileCheck=false
CompressArchives=false
UseCompactedBackupFolder=false
AutosaveConversionLogs=true
UseStartupGroup=true
AutoSaveCurrentCategory=true
ConsolidationAlgorithm=AllSequential
ESLCountsReflectAnticipated=true

[Volatile]
CurrentModSet=-6
CurrentCategory=ALL
IsCustomizedSet=false
LastGame=Skyrim SE
LastSSEErrorLogCheck=637701238109977456
LastTESVErrorLogCheck=0
SaveLootReportInfo=true
SwitchFile=NONE
SwitchDirectory=NONE

[LootReport]
Path=D:\Developing\Projects\IMT\ModSetManager\ModSetManager\bin\Debug\Data\loot.json
ReadComplete=false
DateUpdated=637672779504798743
YPos=20515

 

Edited October 18, 2021 by primem0ver

[primem0ver]

  On 10/17/2021 at 9:28 PM, 0Outcast said:

I personally only tinkered with Wrye Bash and zEdit. But such a tool would be Godsend.

The time though... To Programm it...Phew. Even a Tutorial Would take a while.

Hmmm... haven't really used those myself to fix errors. It doesn't replace tools (at least not yet... see below), it just makes them easier to use.

 

To fix errors, it gives you an interface (launched from ModOrganizer) that shows all mods with errors and color codes the errors to give you clues on how to fix them. The tool has buttons on the top that help you launch the tool you need act the click of a button (loading ONLY the mod to fix and it's masters). It takes care of all the details necessary to correctly launch the tool without using the interface (except for Creation Kit which unfortunately doesn't allow you to automatically load a mod).

 

It also allows you to track what you did to fix the mod (in a very general sense... the "Mark as..." buttons in the UI below). Here is what it looks like:

 

 

As I have used SSEEdit to clean files, I have realized that it has some significant weaknesses that are annoying (such as no option to search for records containing text patterns) AND to help achieve the useful goal that gnarly1 proposed here, I am seriously reconsidering expanding this tool to replace much of the functionality of SSEEdit when it comes to fixing errors. Gnarly1's idea is very intriguing. Imagine being able to view record conflicts the same way you can view file conflicts in ModOrganizer (color coded on the scrollbar). I think that would be VERY useful in figuring out how to make simple fixes to resolve record conflicts which can lead to in-game problems. Since my utility already has to read through most records in a file, I decided to add the ability to track records... however this is something I will probably not prioritize so it may take a year or more to fulfill. Creating the record structure is a HUGE task unless I can convince the makers of xEdit/zEdit to share theirs and somehow import it to .NET. Most of the time already spent on this project was to give it the ability to read and change records without errors. Expanding significantly to track each type of record and sub-record will probably take another similar amount of time.

Edited October 18, 2021 by primem0ver

[Deleted120957248User]

  On 10/18/2021 at 2:38 PM, primem0ver said:

 

  On 10/17/2021 at 9:28 PM, 0Outcast said:

I personally only tinkered with Wrye Bash and zEdit. But such a tool would be Godsend.

The time though... To Programm it...Phew. Even a Tutorial Would take a while.

Hmmm... haven't really used those myself to fix errors. It doesn't replace tools (at least not yet... see below), it just makes them easier to use.

 

To fix errors, it gives you an interface (launched from ModOrganizer) that shows all mods with errors and color codes the errors to give you clues on how to fix them. The tool has buttons on the top that help you launch the tool you need act the click of a button (loading ONLY the mod to fix and it's masters). It takes care of all the details necessary to correctly launch the tool without using the interface (except for Creation Kit which unfortunately doesn't allow you to automatically load a mod).

 

It also allows you to track what you did to fix the mod (in a very general sense... the "Mark as..." buttons in the UI below). Here is what it looks like:

 

 

As I have used SSEEdit to clean files, I have realized that it has some significant weaknesses that are annoying (such as no option to search for records containing text patterns) AND to help achieve the useful goal that gnarly1 proposed here, I am seriously reconsidering expanding this tool to replace much of the functionality of SSEEdit when it comes to fixing errors. Gnarly1's idea is very intriguing. Imagine being able to view record conflicts the same way you can view file conflicts in ModOrganizer (color coded on the scrollbar). I think that would be VERY useful in figuring out how to make simple fixes to resolve record conflicts which can lead to in-game problems. Since my utility already has to read through most records in a file, I decided to add the ability to track records... however this is something I will probably not prioritize so it may take a year or more to fulfill. Creating the record structure is a HUGE task unless I can convince the makers of xEdit/zEdit to share theirs and somehow import it to .NET. Most of the time already spent on this project was to give it the ability to read and change records without errors. Expanding significantly to track each type of record and sub-record will probably take another similar amount of time.

 

It will help mod development alot. There is many times you can lose or are unable to track your own progress (development and modification changelog) this can also help Big projects with modding teams and sharing of your activity with a senior member for example that supervises a certain team within a Project like Beyond Skyrim team for example. I know they would benefit alot from it because it just saves so much time if you can have a sorted structure of development changelogs and records. Easier planning and management.

[primem0ver]

Good news (with a potential caveat).

 

I believe the most difficult and time consuming parts of this project are finished. I spent the last two days troubleshooting "cluster" conversions which is basically the final major piece to this project. Cluster versions are a way of handling conversion of master esps (and their dependencies) to esls. If one converts the ids of a master mod in order to convert it into an esl, the ids in the other files in the "cluster" (dependencies) must be changed as well. I successfully converted my first "cluster" tonight without any errors (checked in xEdit). All data on conversions is saved so that any dependencies added to your load order later (such as patches) can use the original conversion data to update the form ids in the patches (and other added mods).

 

So... the next thing is getting this up and ready to put on Nexus. Because of the timing, I have a predicament that I thought I would put on here to get feedback about as I know some of you are eagerly awaiting this project. I have a very busy holiday season coming up; which means I WON'T be able to fix bugs quickly for sporadic periods of time. This will be particularly true during the week and a half from Christmas to New Years. So my question is this. Do you want me to try and get this up before the Holiday season (approximately at the end of November) even if it means I may not be able to address issues right away? Or wait until January?

[VulcanTourist]

  On 11/12/2021 at 4:57 AM, primem0ver said:

So my question is this. Do you want me to try and get this up before the Holiday season (approximately at the end of November) even if it means I may not be able to address issues right away? Or wait until January?

I somehow missed the e-mail notification for this comment back in November when my reply might have made the most difference, but...

 

... I think I'd be most happy if you delayed release until when you have time to deal with any initial showstoppers that might crop up. I haven't programmed in decades, so as strictly a user at this point I'd be reliant on you/others to fix those issues, and after so much anticipation it would be hard to set it aside then to wait for those issues to be addressed weeks later. It's easier to wait until you'll have time to spare and we can help you discover any issue and get rapid feedback.

 

Enjoy your holiday! And have ya heard about Bethesda's new Starfield RPG? A whole new RPG to mod!

Edited December 13, 2021 by VulcanTourist

[primem0ver]

Update:

 

Ugh.... problems problems problems. In building my mod set with this utility, I discovered Skyrim's max reference bug/limitation. This probably led to some of the crashes I would experience when playing in previous games. So I am currently adding the ability to deal with this issue into my utility, for those who don't mind dealing with the potential side effects. I am guessing it will be at least March before this is ready.

Edited January 21, 2022 by primem0ver

[VulcanTourist]

  On 1/21/2022 at 1:46 PM, primem0ver said:

Update:

 

Ugh.... problems problems problems. In building my mod set with this utility, I discovered Skyrim's max reference bug/limitation. This probably led to some of the crashes I would experience when playing in previous games. So I am currently adding the ability to deal with this issue into my utility, for those who don't mind dealing with the potential side effects. I am guessing it will be at least March before this is ready.

Any good news to report?

[VulcanTourist]

  On 1/21/2022 at 1:46 PM, primem0ver said:

Update:

 

Ugh.... problems problems problems. In building my mod set with this utility, I discovered Skyrim's max reference bug/limitation. This probably led to some of the crashes I would experience when playing in previous games. So I am currently adding the ability to deal with this issue into my utility, for those who don't mind dealing with the potential side effects. I am guessing it will be at least March before this is ready.

  On 4/3/2022 at 7:35 PM, VulcanTourist said:

Any good news to report?

 

All this silence is making me wonder if I'm ever going to enjoy a no-compromise solution to my too-many-mods problem and enjoy Skyrim again without the pain and agony of re-evaluating my modding from scratch.... *sob*

Edited April 17, 2022 by VulcanTourist

[primem0ver]

I have built my first load order with my utility. My load order consists of around 1700 plugins (around 400 of which are merged). I may have a few more issues to resolve as I have to run a few more tests. Then I have to get the bugs out of the software... but a REALLY rough draft is nearly complete. I have had to add some MAJOR time-saving features out of necessity, the best of which I finished today (after 16 hours of work). I have spoken some with the developer of zEdit/zMerge. He had this to say about my software... when I showed him the picture below a few days ago (showing a different feature I added about half a week ago). I had to ask him a few questions about the files zMerge saves in order to get it to work.

 

  Quote

 

damn that's amazing ! i think you will get a lot of people praising you if you release this. I'm saying this not because I think it should be a motivating factor - but just to give you a heads up this solves for a real pain point

 

 

Here is a sneak preview of the features (a draft of the Nexusmods page that will go with it). Please be patient though. I have 2-3 major hurdles to get through before I can post it. I have to fix the master converter, the cluster converter, and keep from having to restart it all the time. A single debug session to figure out the simplest problem often takes more than 10 minutes (because loading while debugging takes nearly 10 minutes if I use the full program). That is the main reason why this is taking so long. Open the spoiler to view the features. (Remember this is a really rough draft... things could change a bit).

 

 

  Reveal hidden contents

 

Mod Set Manager Tool Suite (And Complete ESL Conversion Suite)

 

Introduction:

Are you tired of taking forever to build a large load order? Or do you skip out on mods because of the time it takes to add them into a working load order? Would you like some cool tools to help? Tired of endless checking and unchecking mods in your mod organizer when the NetScriptFramework log doesn’t help you find the cause of a crash?

This is the Quintessential Dream Tool for “mod-aholics” like me. It is also a helpful tool for converting mods to ESLs and Masters, managing saves, and keeping notes on mods.

There are lots of steps and tools that are part of the workflow and process of building, maintaining, and updating a load order of working mods. This tool acts as a mediator between separate tools to facilitate the process of building and testing a load order so that it is much quicker, and easier to accomplish. It covers everything from installing and tracking mods, keeping notes on them, converting mods to ESPFEs (ESLs), building and managing merges, EASILY resolving merge conflicts, merging load orders, managing characters and saves, as well as providing at-a-glance feedback when crashes occur to find out possible mods involved even when they are part of a merge.

This is the culmination of giving up 90% of my free time for more than 2 years. I hope you find it useful.

 

Description:

Mod set manager was designed to make it easy to organize and build and test large load orders quickly. It offers a full suite of ESL conversion options and utilities AND integrates with Zmerge, xEdit, and other utilities to make sure you have working load order. You can convert ANY mod that is valid for ESL conversion to an ESL instantly. Or mass convert them all!* This includes voiced mods, NPC mods (with face tints), and mods with dependencies (opposite of masters), all at the click of a button. It can convert about 10 mods per second (Just under 600 per minute) on a factory boosted i7 8700 (no custom overclocking). It backs up any changes made and offers a wide range of documentation options, using an Excel spreadsheet document. (You DO NOT need Excel to use MSM). The processes of cleaning and fixing errors has also been streamlined to a simple point and click interface. There are a total of three utilities included. Testing is best done with some tools I recommend below; however, MSM gives you immediate and direct feedback when a crash occurs (as long as it is loaded).

*note, mass conversion only converts eligible mods WITHOUT mod dependencies. However, there is the “cluster” converter when you want to convert mods that have dependencies (or have masters that can be converted).

 

**IMPORTANT ALPHA NOTES:
1. This mod is in alpha testing. This means it is a bit buggy and quirky. However, it is fully functional, and for the most part, it works just fine. I have converted over 1000 mods to ESLs successfully but on occasion there might still be a failure. If the mod fails to convert during the conversion process, it will be replaced with the backup automatically. Effectively that means the original version remains intact. If the mod has errors that didn’t exist before conversion, then you can go to the conversion history and restore the original version.

2. Please be patient with me in regard to troubleshooting and waiting for updates. I spent the last 2½ years working on this project, building my first SE load order for the purpose of getting this project right. I am going back to school to get my masters AND have a full time job. So it may take time. If anyone is good at C# and wants to help, by all means, let me know.

3. The main interface showing mods has LOTS of visible information. It might be a bit overwhelming. However, most of the information is either informative for people who want to know why a mod can or cannot be converted to an ESL; or at least somewhat useful for the purpose of categorizing, annotating, and knowing stats about any particular mod. If enough people request it, I may make it so columns can be customizable (meaning you can show or hide certain columns).

 

How it works:

There are many auxiliary functions. However, the core of Mod Set Manager functionality is based on it keeping a “master” version of the mod list, load order, and plugin list files along with an excel spreadsheet in your mod organizer/vortex profile to keep track of the user-modifiable data associated with each mod. That way the load order can be manipulated on the fly to perform tasks which are limited to a specific number of non esl mods (such as checking for errors in xEdit). This can be done using tags (official “sets”), categories, selection, and search filters.

 

The MAIN Tool:

Mod Set Manager – which I often refer to as MSM. This is the main tool which provides a LOT of functionality and includes the manager and conversion suites.

 

The Three Additional Tools:
The Cleaning Tool – Steps you through each mod that requires cleaning and handles the entire process of cleaning a mod at the click of a single button (Requires exporting a list using Loot and integrates with xEdit’s autoclean feature). This process CAN be automated but keep in mind that it isn’t always recommended.
The Fixing Tool – Steps you through each mod with errors and gives you a suite of tools that you can click to (almost) instantly load the mod and fix it (Integrates with xEdit, xEdit autoclean, and Creation Kit – REQUIRES xEdit)
The Testing Tool – When you can’t find the cause in the crash information that is graphically displayed for you (if you have NetScriptFramework logs to parse), you have this tool to help you. It automatically troubleshoots your load order for you. All you have to do is press the launch button. When you close Skyrim, or it crashes, just report the result and it will automatically pick which mods to test next. You can modify this list if you wish. The list of mods to load is automatically generated. You can also use this tool to quickly find incompatible mods.

 

 

**Core Features**:

· Simple yet informative, advanced color-coded table interface to help you find information you are looking for at a glance.

· Batch convert groups of mods or your entire load list with minimal interaction

· FULL FEATURED ESL CONVERTER: Instantly convert nearly ANY mod (with fewer than 2048 internal records) to an ESL WITHOUT errors (retaining FULL functionality):

o Regular mods (any mod without a defined cell or an explicit form id inside a script)

o Quest mods (including mods with an SEQ file)

o NPC mods

o Most Voiced NPC mods (see important note for exceptions below).

o Mods with Dependencies (automatically handles dependencies with your approval)

o Mods with BSAs

o IMORTANT NOTE: mods that add dialog to characters in a master (such as mods that add dialog to characters in other mods or DLCs such as Serana dialog mods) are not currently supported. I hope to add this feature in the future.

· ESL Convertibility is automatically detected and color coded so that you know at-a glance which mods can be converted and what it will take. (Key: green can be directly converted to an esl; yellow requires id compaction; purple means that it can be converted directly but masters can be converted as well which will require id changes for master references; orange is the same as purple but the mod will require id compaction; red means that it is a master with dependencies and all dependencies must have their references to master id’s changed to the new ids.)

· Keeps a record of all ESL conversions and the newly assigned ids. This means that if you download a patch or some other mod that uses the converted mod as a master, [NEED TO IMPLEMENT]: MSM will notify you that ids need to be changed in the newly downloaded mod. (Or optionally, you can convert it back to its original if it isn’t already committed… see last point below). [NEEDS TO BE IMPLEMENTED]: Optionally, you can keep a backup of these records.

· Keep yourself from accidentally converting mods that don’t convert well by adding them to the “Do Not Convert” list. (SkyUI and FNIS come pre-added to this list).

· Create and manage merges using tags in the same MSM interface without having to load zMerge. (You only use zMerge to perform the actual merge).

• Easily find all merge conflicts and fix them with a couple clicks of a button (For example, a mod’s master is in a merge that the mod doesn’t belong to).
• Full Featured Save-game Manager. (This came about because I was angry that even with the new “save” features and id system, Bethesda did not include managing features with the game). Individual saves are grouped into games (characters). You can delete individual saves OR delete an entire game/character all at once from a simple interface.
• Integrated NetScriptFramework crash-log analyzer and GUI based feedback that identifies mods which are potential causes of crashes even if the mod has been merged.
• If you can’t find the cause of the problem using the crash-log analyzer, the mostly automated Plugin Testing Tool can help you figure out which mod is causing the crash MUCH quicker than endlessly checking and unchecking boxes. NO LOAD/PLUGIN ORDER SAVES NECESSARY!!!
• Keeps accurate track of the exact number of references your load order will use (NOT a ballpark figure like xEdit scripts).
• Identify and optionally select any mod that is a dependency of the selected mod (i.e. a mod which has the selected mod as a master). Think of a dependency as the opposite of a master.
• Find and optionally select a mod that meets almost any criteria (using the Filter option)
• Option to convert mods to masters. While this isn’t guaranteed that it will work, it is more effective than the mentioned scripts. (Some mods just can’t be converted using anything but versioning in the creation kit).
• Backs up any converted masters so that they can be replaced with the original at the click of a button.+

· Keep custom comments, titles, and info on mods in seven custom columns used for this very purpose.

· Categorize, sort, filter, and select mods for easy manipulation

· Step through or batch clean mods (based on pre-selection… requires LOOT).

· Step through each mod with an “Easy loader” interface for cleaning mod with errors that can’t be fixed through the traditional xEdit clean method. (Requires loot)

· Nearly full featured conversion history interface that allows undoing conversions at the click of a button.

· The same conversion history interface allows you to clear up disk space using a “commit” feature once conversions have been tested In game (deletes backups).

 

Requirements:

• Mod Organizer (A Vortex version is planned but I need C# volunteers who use Vortex for a conversion. Please message me!)
• xEdit for error scanning, and optional cleaning functionality

Required for Full Functionality:

• zEdit/zMerge for merge functionality (which was the original purpose for MSM’s design). Without this utility you can define merges but not actually build them.

Highly Recommended:

• Modsvaskr (and all its requirements) for Full Featured, non-spoiler testing functionality. IMO, this is an essential tool for a new large load order. It is a bit tricky to set up but it makes sure it works before you begin a game. I highly recommend this for people using MSM to build large load orders because it is highly likely YOU WILL run into problems/incompatibilities/CTD’s for large untested mod sets. MSM feedback from crashes caused when using this utility will give you the quickest path to a fully functional and semi-compatible load order.

 

Installation:

1. First, install the requirements.
2. Install MSM using the installation wizard downloadable from the files page.
3. You can use the zipped portable version, but this requires you to configure required information manually.
4. DO NOT RUN THE MAIN MSM APPLICATION THROUGH MOD ORGANIZER. If you plan to use the 2 additional/non self-contained tools, you need to configure them in Mod Organizer using the traditional method of tool configuration.

 

Running the Software:

• Normal use is straightforward
• There are a few command line options which are explained in the documentation.
• There are also several documented settings that can help you customize your experience. Clicking on them in the settings dialog will tell you what they do.
• There is one set of settings that are not documented but are only intended for use if making publically available videos/pictures of the software. Their purpose is to disguise those mods that you have installed from… ahem… less reputable sites. For those settings you will need to manually edit the MSMsettings.ini file.

 

** Troubleshooting and Error/bug reporting **

• PLEASE REPORT ALL ERRORS WHEN A MOD FAILS TO CONVERT TO AN ESL. Please include the name of the mod and the esp that failed. I WILL ignore any reports that do not include this information. Attaching the conversion report is also very useful.
• IF XEDIT REPORTS ANY ERRORS THAT DID NOT EXIST BEFORE CONVERSION, PLEASE REPORT THESE AS WELL. If they existed before conversion, don’t bother (should be obvious). Please go into MSM after scanning for errors and pull up the mods internal error report. Copy the text there and include it in the report.
• If MSM fails to start (NO SPLASH SCREEN), chances are you messed with the ini settings and created a problem while doing so. Check for duplicate values (values defined twice in the same section).

 

 

 

Edited July 10, 2022 by primem0ver

[VulcanTourist]

1700 mods! I was stuck at a mere 700, having exceeded the plugin limit and not having the skill to manually ESL-ify any more than I already had. Now you have me genuinely excited and impatient!