Due to Bethesda forums going to a different forum ( Bethesda.net ), reproducing an important topic here on Nexus just in case all old topics fall off the face of the internet as Bethesda move on, and the old locked forums are no longer maintained.
Original research and credit to Lojack and Wrinklyninja, I am just copying the info to preserve it on Nexus and AFK Mods
BSAs and You
With the latest Beta patch for Skyrim officially getting sent out to all clients, there have been some changes to BSA Load Ordering that you might want to know about. Basically, the issues brought up in the previous thread have been fixed. So this post is only here for educational purposes, in case you're curious as to the order of resources loading.
To summarize, the order resources are loaded is:
Registered BSAs -> Plugin BSAs -> Loose Files
With Registered BSAs being loaded in the order listed in Skyrim.ini, and Plugin BSAs loading in the same order as their plugins, which are loaded in the order they are listed in plugins.txt.
Disclaimer: The game doesn't actually work exactly like this. This is how it logically works. As in, if you had to describe the decision process the game goes through, this is it. Realistically, working this way would be a waste of resources and processing time (why load up every file, when you already know a different one is going to "win"?). It's just much easier to understand if you think of it like this. A good description of how it would actually work is here.
- Resource Load Order
- Registered BSAs
- Plugin BSAs
- Loose Files
- What This Means For Mod Users
- Other Questions
- Replacing Scripts
- Directory Thrashing
- What are all these terms you use?
- What about Nitpick?
- Addendum: BSA Compression
1. Resource Load Order
Skyrim loads its resources from the Data folder. Makes sense. These resources can also be packed up into a nice neat little archive, called a BSA. These are Bethsoft's custom archive files, and internally they mirror the layout of the Data folder. They're nice, because they keep the Data folder uncluttered, and it makes it really easy to remove every file from one mod.
Ok, but what happens when the same resource exists in more than one place? For example textures/dungeons/imperial/battlemap01.dds, which is the texture for the map you see all the time in castles, with the little flags and stuff sticking out of them. This file exists in Skyrim - Textures.bsa. That's the original file. If you download a texture replacer, you'll get it as a loose file probably, and if you download HD Skyrimmap from Steam Workshop, you'll get that file in a BSA. So which file actually gets used? Well, which ever one is loaded last. The game loads resources in three stages, with the last files loaded being what you see in game:
Needs Investigation: With pre-1.4.26 Skyrim, only Registered BSAs and Loose Files were loaded when the game first loads. That means anything in a Plugin BSA won't take effect yet - but you'll only notice that if you're trying to replace a Main Menu asset, like the logo. Once you start a game, either by loading a saved game or starting a new one, then the Plugin BSAs are loaded. So in the example of Brumbek's Main Menu Logo replacer, if it's loaded via a Plugin BSA (like it is when installed via Steam Workshop), you won't see the spinning logo until you load a game, then exit back to the Main Menu.
So the investigation needed: Is this still the case? I'll get back to you on that once I get the time to test it.
First the game reads your Skyrim.ini file. There's a section that tells the game which BSAs to load. It looks like this for most people:
[Archive] sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa
The game will load the BSAs in order as listed, BSAs in sResourceArchiveList first, and sResourceArchiveList2 second. So if you manually register a texture replacer BSA (like the HD Texture packs), but put it at the beginning of the list - they won't load! Because the Skyrim - Textures.bsa will override them of course. If you put it after, or at the end of the list, or in the second entry, then they will load.
Easy enough, but there is one drawback: There's a 255 character limit on each of these entries. This is because the game internally uses a 256 byte character buffer to receive the value when reading it. One byte is reserved for the NULL terminator (a byte with a value of 0, indicating the end of the data), so that leaves 255 characters for file names. If you want to get around this limitation, you'll need to use Nitpick.
Next the game loads any BSAs associated with a plugin. When the game loads an active ESP, it also looks for a BSA of the same name (For Oblivion, there were naming conventions that allowed a single ESP to load multiple BSAs in this manner - this is not the case for Skyrim). So dummy.esp will make the game load dummy.bsa. This is true even if the ESP does absolutely nothing (for example, the official HD Texture packs do this, as well as Oblivion's Shivering Isles expansion). Cool! We found an easy way to make our replacement textures only load when we want! Just deactivate the plugin and they don't load, activate it and they do!
Ok, but what happens if two Plugin-loaded BSAs have the same file in it? Well, it's easy now: whichever one loads last wins (as usual). But now there's a reliable way to control which order they're loaded in: they're loaded in the same order as their associated plugins. This means if dummy.esp loads before another.esp, then dummy.bsa will also load before another.bsa, meaning of course that another.bsawill win, because it loaded last.
Sounds good, but I seem to remember there being other issues with using BSAs...
Yes, nothing major really, unless you're trying to mod your game with a lot of plugins.
Due to the way the game indexes plugins and stores the data in your save game, you can only have 256 active mod files. That's because objects are referenced by what's called a FormID, which is a 32-bit (4-byte) unique number, usually written as an 8-digit hexadecimal number. The first two hex digits are used to identify which mod the data came from. Two hex digits can represent a maximum of 256 unique numbers (0-255).
Now, Skyrim.esm and Update.esm will always fill up two of those. Another one that's not obvious is your saved game itself - the one in use is always assigned the ID of hexadecimal FF, or 255. That leaves room for 253 other plugins.
And here we get to our point: dummy plugins that don't do anything at all, but exist solely to make the game load a BSA? Yep, those take up one of our plugin slots. This might not seem like a big deal, but once you start trying to play with 300+ mods, you'll be faced with the hard decision of deactivating some mods. Luckily, dummy plugins you can just unpack the BSA and use the resources as Loose Files, then you don't need the plugin anymore. Other plugins can be merged via Wrye Bash into the Bashed Patch, further cutting down your mod count. And hopefully an equivalent of TES4Gecko will come out eventually, which will be able to merge mods into a single plugin. This last one however has an impact on your saved games, so it's for the more advanced user only.
Needs Investigation: Another small strangeness associated with Plugin BSAs, at least with pre-1.4.26 - they aren't loaded until you load either a saved game, or start a new game. That means if you're still at the Main Menu for the first time, any replacers loaded this way haven't taken effect yet! Woah! Not a big deal, unless you're trying to replace the Main Menu Logo... Oh wait. That mod works fine when installed as Loose Files, or if the BSA is registered in your Skyrim.ini, but it only works as a Plugin BSA after loading a saved game. This behavior still need to be verified now that 1.4.27 Skyrim is out.
This one's simple. If the file exists in the Data folder, it will always win. It will always be used, even if the same file exists in a Registered BSA or Plugin BSA. This is how most texture replacers work, and this is what mod managers like Wrye Bash or NMM are made for. Managing the Loose Files so you don't have to track them yourself. It's easy to keep track of which file came from which installer when you only have 3 mods. But when you get up to 150 mods, each with their own resources, a lot of them overlapping, a mod manager just takes the headache out of it.
2. What This Means For Mod Users
Not much really. There is one minor disadvantage to all this: Steam Workshop uses BSAs by default, even for texture replacers that don't need a plugin. That means you end up with an ESP that is only there to load a BSA, taking up one of your 253 mod slots doing it. If you start getting close to the 253 mod limit, you can cut back by either:
- (Recommended) Unpacking those BSAs to Loose Files. A useful tool for unpacking the BSA is DDSopt. Once the BSA is unpacked, just pack up those Loose Files along with any documentation into an archive, and install with your favorite mod manager. Now you can deactivate the plugin, saving you a mod slot.
- (Not Recommended) Register the BSAs. This method you'll use your Skyrim.ini to load the BSAs instead of using a plugin. First you'll need to deactivate or delete the plugin. Next, you'll need to add the name of the BSA to your Skyrim.ini, in the following section:
[Archive] sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa: The reason I don't recommend this method is that it's harder to change the order. This method you need to edit your Skyrim.inieach time you want to change the order, whereas with Method #1, you just either change the Installer Order of the package in BAIN, or in NMM. Also, you'll run up on the 255 character limit pretty quick, which means that you'll need to make use of Nitpick, which in turn requires SKSE. There's nothing wrong with requiring SKSE, but you'll have to wait for a SKSE update after each game update.
Again, each of these tricks only works if the plugin in question is an empty, dummy plugin, solely there for the purpose of loading the BSA. You can check this using TES5Edit or another similar utility, or right click on the mod in Wrye Bash and select 'Mark Mergeable...' to find out if the plugin is empty. Wrye Bash will tell you if the plugin is empty, but it won't say anything at all relating to its "emptyness" if it's not empty.
3. Other Questions
Pre-1.4.26 for Skyrim, you couln't replace a Vanilla Script file with one in a Plugin BSA. This is no longer true, it has been fixed, and script replacers should work as both Loose Files and Plugin BSAs now.
What is it? link. Basically, simply having too many ESP, ESM, and BSA files (even inactive) in the data directory would cause Oblivion to go crazy, bringing your game to a crawl as disk I/O went through the roof. It's why Wrye Bash has the Auto-Ghost feature.
Does it affect Skyrim? Sort of, but not in the old sense of the term. There's a different problem with Skyrim, such that if you have over 508 mod files in your data directory, the engine just plain refuses to load any of them, active or not. To fix this, use Nitpick. Wrye Bash's ghosting will also work here since, Wrye Bash will not allow more than 255 active plugins at once, and ghost the rest, meaning you will at most have 255 files in the data directory with .esm or .esp extensions.
There's also the added problem of plugin INI files (which pretty much no one uses - no fix for this), and part of the old thrashing problem, where the game tries to read inactive plugins. For the last issue, again Wrye Bash's Ghosting feature still works good.
What are all these terms you use?
I used a bunch of terminology here, most of it old, but there's a couple new terms I made up for this explanation.
- Plugin - any .esp or .esm file.
- ESM - Elder Scrolls Master. Usually saved as a .esm file, which has the same format as a .esp file, only one bit (as in, one bit of a byte) is different.
- ESP - Elder Scrolls Plugin. Most mods are ESPs, but some are ESMs. Their format is the same, but the game handles them slightly differently in some specific cases.
- ESS - Elder Scrolls Save. Your saved games. This is loaded after all the ESPs and ESMs are loaded, so that its changes (like what items are in which containers) can override what the plugins say.
- BSA - Bethesda Softworks Archive. Sort of like a .zip file, but specific to Bethsoft's games, and optimized (sort of) for their specific usage.
- Load Order - The order that your mods (ESPs and ESMs) are loaded. When conflicts occur, the last loaded conflict "wins".
- Install Order - The order that files are installed into your Data folder. When multiple packages install the same file, the last installed package "wins".
- BSA Load Order - The order that the game engine loads BSAs. When conflicts in data files occur, the last loaded BSA "wins".
- Plugin BSA - A BSA that the engine loads because an ESP/ESM with the same name is active.
- Registered BSA - A BSA that the engine loads because it's listed in Skyrim.ini.
- Loose File(s) - A resource file in the Data folder that's "loose", as in not packaged into a BSA.
- FormID - A unique ID assigned to each "record" of a plugin. Each object in the game is defined by a record, and each record has a FormID. A FormID is 32-bits, with the first 8 bits defining which ESM or ESP the record belongs to.
- SKSE - Skyrim Script Extender. A nice utility that everyone should be using. It aims to extend the functions available to scripts written for Skyrim. It's also useful as DLL loader, if one wants to inject some code into Skyrim's game engine.
- SD - Script Dragon. Another DLL loader, but with a different goal in mind than SKSE. Both can work along side each other.
- Vanilla - A term referring to Skyrim as distribute by Bethesda and Steam. This means no modifications have been done. Alludes to the "plainness" of the vanilla ice-cream flavor.
What about Nitpick?
I'm still maintaining it. shadeMe may also release updates as well, he has access to the Skyrim Nexus page, so either he or I can update it. The Nexus page also has links to the source code, but here's shadeMe's source and my source, links in case the Nexus is down at the moment.
4.Addendum: BSA Compression
This question comes up often enough to deserve an answer here: What's better, compressed BSAs or uncompressed BSAs?
To quote Ethatron:
So lets see a little calculation for two operations of reading data from disk, uncompressed and then compressed:
- Harddisk-throughput: 20MB/s
- Memory-throughput: 2000MB/s
- zlib-throughput: 80MB/s
- uncompressed size: 100MB
- compressed size: 60MB
In case the data is very small it's probable that seeking time will start dominating the times:
- Reading 100MB at 20MB/s from disk takes 5s.
- Reading 60MB at 20MB/s from disk takes 3s, decompressing 60MB at 80MB/s in-memory takes 0,75s, so a total of 3,75s.
Seeking may be as much as 0,002s. Nevertheless it doesn't change the fact that reading compressed data is faster as long as the decompressor is faster than the disk. It's a rather theoretic notion, but when using compressed solid archives even the seek-times go down as the disc-head has to reposition a smaller distance while bulk-reading from the same archive.
- Reading 1MB at 20MB/s from disk takes 0,05s.
- Reading 0,6MB at 20MB/s from disk takes 0.03s, decompressing 0,6MB at 80MB/s in-memory takes 0,0075s, so a total of 0,0375s.
Also, in reality with a modern system the decompression speed of zlib reach realistic 250-500MB/s (as you can see here), which can be far beyond any SSD-speed.
So for the most part, compression is good for you. There are a few cases where this might be detrimental though: At very high compression rates, it seems some files are affected - notably sound files seems to take on a "fuzzy" sound to them in many cases. Textures and Meshes should be fine going to max compression, but be sure to test things if you're going to up the compression on your BSAs.
Edit by Alt3rn1ty typo's plus .. : For reference as to what not to compress at High Compression levels - See this topic first post ( Any original BSA flagged as UnCompressed by Bethesda, is a pretty good bet contains 'some' files ( not necessarily all of them ) which should not be compressed )
Credit to Fireundubh on STEP forum for finding out the spoilered information