SSE Lessons Learned about the Creation Kit

[exsomet]

Hi all.

 

I've recently been trying to come up with a better workflow for myself in regards to my modding environment. I wanted to share what I've come up with in the hopes that a few people might share some feedback on improvements. I'm also hoping that by typing this up and sharing it, I can cement some of the things I've discovered and help myself improve.

 

My motivation for doing all this was to learn more about the Creation Kit and Mod Development in general, as well as to break out of the clunky workflow I had going that used mostly default tools from the Creation Kit.

 

Fair warning: This is going to be a long post about stuff that most modders probably already know. Skip to "Solution" below if you don't want the background.

 

Workflow Problems

As far as mods go, most of what I do is fairly simple. It lives on my computer, never gets published, and eventually gets uninstalled and fades away, or replaced by something someone else did that does the same thing better. The result of that is that I pretty much use the default tools and workflow for anything I do. Scripts are created using the Creation Kit, edited using the Creation Kit editor window, and saved to the default %skyrim%\Data\scripts\source\ directory. As a programmer, this has always sort of bothered me, but until recently it was functional and so I was happy. I'm overdue for a PC rebuild, so I thought I'd take this one's last few breaths and try to hammer out some solutions to a few of the things I take issue with - worst case, if something got screwed up, it wouldn't be that big of a deal.

 

Script Management

The first gripe I have has to do with script management in the Creation Kit. Scripts created through the editor are all dumped into one place. This makes tracking them, as well as managing them and cleaning them up, a real pain. It's really easy to have your source directory go kind of wild with scripts that aren't even being used.

 

Source Control

My second issue was that I didn't see a very clean way to use source control. Since scripts are all dumped to %skyrim%\Data\scripts\source\ (the same place as the vanilla scripts), the options seemed kind of limited.

• I could make the entire \source\ directory a git repository. This means I would have to .gitignore all of the vanilla scripts (or add them to source control, which seemed pointless). This also means if I made two mods, I would need to either create two branches in the repository which track two different projects, or make the one location on my PC (\source\) actually be multiple repositories by deleting git-specific files and re-initializing the repository. Neither of those is really how git is intended to work.
• I could not use git. Yuck.

I poked around a bit on various sites, and it does seem like the Creation Kit has some sort of integration with Perforce, but it also seemed to be really painful to get set up properly, and involve learning an entirely different way of doing things that what I'm used to with git, which I'm not really interested in doing.

 

External Editor

Papyrus Scripts are easy enough to edit with an external editor in their default state. File -> Open, and you're there. The problem is that a lot of the nifty features I use an editor for (VS Code, so tasks, debugging, extensions, etc) rely on being in a defined work space (project) and are customized for that work space. As with git, I could make the entire \source\ directory be my work space, but the customization for one mod falls apart as soon as I start working on another. This once again makes it a pretty big pain point.

 

Mod Organizer

A lot of my complaints seem to be addressed by Mod Organizer to some degree. In a nutshell, it looks like Mod Organizer creates a virtual file system which lets you have directories outside of %skyrim%\Data\scripts\source\. While that solves the management, source control, and editor customization problems, it also requires that I adopt a new program, abandon one I've used happily for years (NMM), and learn a new workflow for using MO which is thinly documented and seems to vary depending on who writes the article and what version of Mod Organizer they're using. As with Perforce, I'm sure it works fine, but I'd really prefer not to use Mod Organizer unless I have to.

 

Early Attempts at Solutions

I made a couple passes at solving a few of these problems and learned quite painfully what was and what wasn't going to be easy to fix.

 

ScriptCompile.bat

For starters, I swapped from compiling scripts through the Creation Kit to compiling them using ScriptCompile.bat. I took the first version I used off of the Creation Kit wiki:

:: Will need to edit this file every time CK updates.
:: Designed to work in conjunction with tasks.json in vscode
:: %1 - Directory of scripts file
"C:\Program Files (x86)\Steam\SteamApps\common\Skyrim Special Edition\Papyrus Compiler\PapyrusCompiler.exe" %1^
 -a -f="C:\Program Files (x86)\Steam\SteamApps\common\Skyrim Special Edition\Data\source\scripts\TESV_Papyrus_Flags.flg"^
 -i="C:\Program Files (x86)\Steam\SteamApps\common\Skyrim Special Edition\Data\source\scripts"^
 -o="C:\Program Files (x86)\Steam\SteamApps\common\Skyrim Special Edition\Data\scripts"

As you can see here, I did make a few changes:

1. Added comments and split it into multiple lines
2. Removed the pause at the end (this is due to the way I compiled it, using tasks.json in VS Code)
3. Passed in a directory to %1
4. Added the -a flag to compile all scripts in %1, rather than just compiling a single script.

The net result was that I was able to run the compiler from the command line. Unfortunately, I couldn't compile scripts that were not in the %skyrim%\Data\scripts\source\ directory (if you're familiar with ScriptCompile.bat, you may already see why). Still, I was happy with that for the time being as it did seem to show some promise. More than anything else, it planted the seed in my mind that I could probably find a way to hammer out some sort of workflow that I would be happy with, even if I didn't know how yet.

 

Git In A Repository Now!

I decided to try on a different problem for size, turning to the issue of source control. I had poked around GitHub and seen quite a few mods on there, so it seemed like it had to be possible to have mod files exist in a directory other than the Skyrim Data root. I wiki-hopped for around a day and a half or so, but nothing really gave me any indication as to what I needed to do to accomplish what I wanted. The closest I came was the Mod Organizer solution (see above), but for the reasons stated I didn't really want to do that. I even got so frustrated as to wonder if people were manually copying files over to track in source control.

 

I decided to give it a shot and see what happened for kicks, so I created a repository and stuck some manually-created .psc files in it, then ran ScriptCompile.bat against it. Immediately, it failed to compile, complaining that the script helloWorld.psc was not found. This seemed strange to me because I was passing the file in to the compiler. Taking a file, and then complaining that you don't know which file I wanted to compile seemed pretty backwards.

 

It took another few hours of digging, but I eventually found out that the files needed to be in %skyrim%\Data\scripts\source\ to be compiled. Even if I called the compiler against files in an external repository, they had to be in \source\ or they wouldn't compile. Ew.

 

This kind of seemed like a dead end, but I decided to poke around the internet a little bit more before I gave up.

 

AdvancedPapyrus

By some luck, I came across Advanced Papyrus. Immediately after stumbling across the GitHub repository, I found about seven posts on different sites that referenced it, which is about how these things go I guess. I set it up and managed to compile a few scripts with it, but for the most part didn't get what it was actually doing for me until I slowed down and re-read the documentation.

 

 

 

A wrapper program that allows for more advanced use of the Papyrus compiler from within Creation Kit.

 

The part I missed on my first readthrough was "from within Creation Kit". Since I was using ScriptCompile.bat, I already had access to all of the command line switches that Advanced Papyrus was exposing, but if you compile from within the Creation Kit, these switches are set to default, unchangeable values. This was one of the first breakthroughs I had. Once I understood that the Creation Kit was importing only the script source directory by default, I understood the correlation between what the compiler is told to compile and where it looks for it. It led me to the realization that I had to import more than just the \source\ directory within the Skyrim data root - I also had to import the directory containing the scripts to be compiled. I'm still not entirely comfortable with this, but it worked after I changed my import line in ScriptCompile.bat to this:

 -i="C:\Program Files (x86)\Steam\SteamApps\common\Skyrim Special Edition\Data\source\scripts";%1^

The difference is subtle, but notice at the end - I've added ;%1. This causes the directory passed in as the first argument to be appended to the end of the import list.

 

I ended up reverting back to the vanilla compiler since I didn't really need Advanced Papyrus, but it gave me a whole new thread to start pulling on.

 

Solution (This is the Interesting Part!)

With a host of new information gained by trial and error in a couple areas, I decided that adopting Perforce and Mod Organizer (workable as they both may be) was not going to be required to build a workflow that I could be happy with. Instead, I set out to define what a good workflow would be so that I could determine how to build it.

 

Requirements and Standards

In defining what I required in a workflow, I tried to mimic what I would expect from a non-Skyrim software development project. This gave me a couple of clear goals/requirements

 

1. I should be able to treat Skyrim mod development like it were any other project.
2. Each mod should have it's own repository outside of the Skyrim data root.
3. I should be able to create scripts from within VS Code (or more importantly, without going through the Creation Kit)
4. I should be able to compile scripts without using the Creation Kit.
5. A mod's repository should have all of that mod's fragments added to it automatically.
6. My mod's .esp should be saved to the repository.
7. When I'm not working on a mod, I don't want it's files in the Skyrim directory (unless it's completed and I'm using it in-game).

With the requirements outlined, I also took the opportunity to define a couple of standards that I'd try to adhere to across development.

• Naming Convention
  • I decided to prefix all scripts with _exso_ in order to make them easy to find. I think best practice says you should use the mod name, but I'm lazy (and while I didn't know it at the time, my workflow now depends on this).
  • I configured the Creation Kit to prefix all fragments with _exf. I would prefer _exso_ on them too, but you're limited to four characters. Such is life.
• Version Conventions
  • Not super relevant to the topic at hand, but I suppose it bears mentioning. I decided on major.minor. I don't really do lots of versions for most mods, so it's kind of superfluous, but meh.
• Repository structure
  • For simplicity, I decided to mirror the Skyrim Data directory structure (for scripts). That is, within a given repository, I create the structure <Repo>\data\scripts\source\ and then place all scripts and fragments in \source\.
  • The .esp is placed within <Repo>\data\.
  • Outside of \data\, I place other things like readmes, documentation, my gitignore, etc.

Workflow and Automation!

With the requirements determined, I outlined what the manual process of developing a mod might look like:

1. Use NMM to activate a profile containing no active mods.
2. Create a repository somewhere.
3. Load CK and master files
4. Create scripts in %repository%\data\scripts\source\, conforming to the defined naming convention (see above).
5. Create fragments if needed as normal, using the Creation Kit.
6. *Copy fragments to %repository%\data\scripts\source\
7. *Run "ScriptCompile.bat <Repo>"
8. Click save in the Creation Kit
9. *Copy <mod>.esp to <Repo>\data\
10. *Delete all script sources with the prefix _exso_ or _exf_ from the Skyrim data directory
11. *Delete all compiled scripts with the prefix _exso_ or _exf_ from the Skyrim data directory
12. *Delete <mod>.esp from the Skyrim data directory
13. Use NMM to restore my normal profile containing active mods.

That's a lot of steps, but it meets all of the requirements above. More importantly, if you look close, there are quite a few of those steps that can be automated (any of them with a *). For doing so, I use batch files. As part of my day-to-day life, I have a folder on my computer called bat which is added to my PATH environment variable. This lets me call any batch file located there by name, without fighting typos to get the full path to a script. So instead of C:\Users\exsomet\Documents\TaxReturns1997\DontLookInHere\youlooked.bat, I can just type youlooked into a command prompt.

 

skbuild

The first batch file I built is called skbuild. This targets steps number 6, and 7. skbuild has a couple of requirements to run successfully:

1. All repositories must be stored in the same place. For example, in C:\Repositories.
2. Mod repositories MUST adhere to the "Repository Structure" standard defined above.
3. Scripts must begin with _exso_
4. Fragments must begin with _exf.

I invoke skbuild by passing in the name of a repository, such as "skbuild temple-healers". The first thing it's doing is to do some validation - have I passed in an argument and does the directory I passed in actually exist. If both of those check out, it does the following, in order:

1. Copy all fragments from %skyrim%\data\scripts\source\ to %repository%\data\scripts\source.
2. Invoke "ScriptCompile.bat %repository%\data\scripts\source. ScriptCompile.bat is configured to output .pex files to the skyrim data folder.

The net result of calling skbuild on a repository is that I end up with all source files (scripts and fragments) in the repository, and up-to-date compiled scripts in the game data root. I can call skbuild at any time during the mod development process, including multiple times in succession. Note that I'll have to run skbuild every time I start working because none of the .pex files will exist in the skyrim data root (as a result of skclean), but that isn't a big deal at all.

 

skclean

The second batch file I built is called skclean. This targets steps number 9, 10, 11, and 12. skclean has the following requirements:

1. All repositories must be stored in the same place. For example, in C:\Repositories.
2. Mod repositories MUST adhere to the "Repository Structure" standard defined above.
3. Scripts must begin with _exso_
4. Fragments must begin with _exf.
5. A file called FileManifest.txt must exist at the repository root (so, next to \data\ but not in it). If I forget to create one, the bat file creates this for me.

I can invoke skclean in the same manner as skbuild, "skclean temple-healers". As with skbuild, it immediately attempts to do some basic (non-exhaustive) validation, with the additional step of validating the FileManifest.txt exists.

 

Once validation checks are done, if no errors are found, skclean does the following, in order:

1. Copy all fragment sources from %skyrim%\data\scripts\source\ to %repository%\data\scripts\source.
2. Delete all compiled (.pex) scripts from the skyrim data root with the prefix _exso_
3. Delete all compiled (.pex) fragments from the skyrim data root with the prefix _exf_
4. Delete all fragment sources from %skyrim%\data\scripts\source\ beginning with _exf_
5. Move all files listed in FileManifes.txt from the skyrim data root to the repository data folder.

The last step (FileManifest.txt) is a band-aid solution I put together after fighting with a bunch of for loops in a batch file for a couple hours. Originally, the idea was to store a list of .esm and .esp files that made up vanilla Skyrim, then loop through and compare every file in the data directory with that known file list. Since that turned out to be a bucket of nope, I settled for just listing the files I wanted to copy per-repository in FileManifest.txt, then reading that file and copying them when I invoked skclean. It's hackish, and if I ever use a lot of files it may get painful, but for now it works.

 

Conclusion and Next Steps

This workflow works well for me for several reasons. It meets all of my requirements outlined above, and It also prevents me from having to adopt new tools (Mod Organizer, Perforce). I also like that it mimics Python to some degree, a language which I use a lot. In python, I write code then use the command prompt to call it. There's a similar feeling here in that I can write scripts, then use skbuild from the command prompt to compile them.

 

I'm considering the bat file method a sort of "proof of concept". I'll use it for awhile and if there are no major issues, it's likely I'll build a better version in the form of a windows service or something that watches, moves, and compiles files automatically instead of having to be invoked. I may also implement further automation around building out repository structure, automatically populating FileManifest.txt, and any other maintenance tasks that crop up.

 

I expect I'll run into issues here and there over time. This workflow may have some problems when I start dealing with larger mods, but for now it's a great first step and should let me get deeper into modding Skyrim for a long time to come - if anyone has feedback on thoughts, potential issues, ideas, or things I may have gotten wrong, I'd love to hear them!

 

Edit:

Batch files located here - https://bitbucket.org/exsomet/skyrim-mod-batch-files/src/047a22f8fd9e85d6f23d887cd5a6654cfd43830b?at=master

They aren't the cleanest, but maybe I'll get around to updating them one day.

Edited January 24, 2017 by exsomet

[UhuruNUru]

Mod Organizer is very well documented, on the official support site, at STEP Wiki.

See my signature, for the links.

GamerPoets, and Gopher have good videos.

 

However it's 32-bit only, and doesn't work with 64-bit.

For 32-bit Skyrim, it's far superior to NMM, but so is Wrye Bash.

MO was made for Skyrim, the only manager actually made first, and foremost for Skyrim.

It works well, now for Oblivion, Fallout 3, and New Vegas.

While NMM is a Jack of All Games, master of None, and more a Steam Workhop competitor.

 

Tannin was making MO2 until the Nexus hired him, to make the new NMM.

The alpha is not a suitable beginners tool, and I don't recommend it for 64-bit.

 

As for the Script Source Folder, the well made mods mirror the folder structure of, the compiled scripts folder.

Do that and you will not, have all files in one place, most tools allow you to create the extra folders, using the "save as ..." functionality of windows.

You may have to select an "ask me where to save" setting.

 

Failing that, you take control, after each save, manually create the folders, and move each file to the location you choose.

 

You should always work on a copy anyway, so having the currently used version separated, from the working one, is no bad thing, however you do it.