Jump to content

Photo

[Update] BSAs and You


  • Please log in to reply
4 replies to this topic

#1

    Mere Morsel

  • Account closed
  • PipPipPipPipPip
  • 3,566 posts

Due to Bethesda forums going to a different forum ( Bethesda.net ), reproducing an important topic here on Nexus just in case all old topics fall off the face of the internet as Bethesda move on, and the old locked forums are no longer maintained.
 
Original research and credit to Lojack and Wrinklyninja, I am just copying the info to preserve it on Nexus and AFK Mods
 
---------------------------------------------------------------------------

 

BSAs and You

 

 
Lojack : "We're back, with an update on the information provided in [IMPORTANT] BSAs and You!

First off, make sure you've read [IMPORTANT] Load Order and You, by wrinklyninja.

 

Spoiler

 

With the latest Beta patch for Skyrim officially getting sent out to all clients, there have been some changes to BSA Load Ordering that you might want to know about. Basically, the issues brought up in the previous thread have been fixed. So this post is only here for educational purposes, in case you're curious as to the order of resources loading.

To summarize, the order resources are loaded is:
 

Registered BSAs -> Plugin BSAs -> Loose Files

 

With Registered BSAs being loaded in the order listed in Skyrim.ini, and Plugin BSAs loading in the same order as their plugins, which are loaded in the order they are listed in plugins.txt.

Disclaimer: The game doesn't actually work exactly like this. This is how it logically works. As in, if you had to describe the decision process the game goes through, this is it. Realistically, working this way would be a waste of resources and processing time (why load up every file, when you already know a different one is going to "win"?). It's just much easier to understand if you think of it like this. A good description of how it would actually work is here.

Spoiler

 

 

Contents:

  • Resource Load Order
    • Registered BSAs
    • Plugin BSAs
    • Loose Files
  • What This Means For Mod Users
  • Other Questions
    • Replacing Scripts
    • Directory Thrashing
    • What are all these terms you use?
    • What about Nitpick?
  • Addendum: BSA Compression

1. Resource Load Order
 

Skyrim loads its resources from the Data folder. Makes sense. These resources can also be packed up into a nice neat little archive, called a BSA. These are Bethsoft's custom archive files, and internally they mirror the layout of the Data folder. They're nice, because they keep the Data folder uncluttered, and it makes it really easy to remove every file from one mod.

Ok, but what happens when the same resource exists in more than one place? For example textures/dungeons/imperial/battlemap01.dds, which is the texture for the map you see all the time in castles, with the little flags and stuff sticking out of them. This file exists in Skyrim - Textures.bsa. That's the original file. If you download a texture replacer, you'll get it as a loose file probably, and if you download HD Skyrimmap from Steam Workshop, you'll get that file in a BSA. So which file actually gets used? Well, which ever one is loaded last. The game loads resources in three stages, with the last files loaded being what you see in game:

Needs Investigation: With pre-1.4.26 Skyrim, only Registered BSAs and Loose Files were loaded when the game first loads. That means anything in a Plugin BSA won't take effect yet - but you'll only notice that if you're trying to replace a Main Menu asset, like the logo. Once you start a game, either by loading a saved game or starting a new one, then the Plugin BSAs are loaded. So in the example of Brumbek's Main Menu Logo replacer, if it's loaded via a Plugin BSA (like it is when installed via Steam Workshop), you won't see the spinning logo until you load a game, then exit back to the Main Menu.

So the investigation needed: Is this still the case? I'll get back to you on that once I get the time to test it.

Registered BSAs
First the game reads your Skyrim.ini file. There's a section that tells the game which BSAs to load. It looks like this for most people:

[Archive]
sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa
sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa

The game will load the BSAs in order as listed, BSAs in sResourceArchiveList first, and sResourceArchiveList2 second. So if you manually register a texture replacer BSA (like the HD Texture packs), but put it at the beginning of the list - they won't load! Because the Skyrim - Textures.bsa will override them of course. If you put it after, or at the end of the list, or in the second entry, then they will load.

Easy enough, but there is one drawback: There's a 255 character limit on each of these entries. This is because the game internally uses a 256 byte character buffer to receive the value when reading it. One byte is reserved for the NULL terminator (a byte with a value of 0, indicating the end of the data), so that leaves 255 characters for file names. If you want to get around this limitation, you'll need to use Nitpick.

Plugin BSAs
Next the game loads any BSAs associated with a plugin. When the game loads an active ESP, it also looks for a BSA of the same name (For Oblivion, there were naming conventions that allowed a single ESP to load multiple BSAs in this manner - this is not the case for Skyrim). So dummy.esp will make the game load dummy.bsa. This is true even if the ESP does absolutely nothing (for example, the official HD Texture packs do this, as well as Oblivion's Shivering Isles expansion). Cool! We found an easy way to make our replacement textures only load when we want! Just deactivate the plugin and they don't load, activate it and they do!

Ok, but what happens if two Plugin-loaded BSAs have the same file in it? Well, it's easy now: whichever one loads last wins (as usual). But now there's a reliable way to control which order they're loaded in: they're loaded in the same order as their associated plugins. This means if dummy.esp loads before another.esp, then dummy.bsa will also load before another.bsa, meaning of course that another.bsawill win, because it loaded last.

Sounds good, but I seem to remember there being other issues with using BSAs...

Yes, nothing major really, unless you're trying to mod your game with a lot of plugins.

 

Due to the way the game indexes plugins and stores the data in your save game, you can only have 256 active mod files. That's because objects are referenced by what's called a FormID, which is a 32-bit (4-byte) unique number, usually written as an 8-digit hexadecimal number. The first two hex digits are used to identify which mod the data came from. Two hex digits can represent a maximum of 256 unique numbers (0-255).

 

Now, Skyrim.esm and Update.esm will always fill up two of those. Another one that's not obvious is your saved game itself - the one in use is always assigned the ID of hexadecimal FF, or 255. That leaves room for 253 other plugins.

 

And here we get to our point: dummy plugins that don't do anything at all, but exist solely to make the game load a BSA? Yep, those take up one of our plugin slots. This might not seem like a big deal, but once you start trying to play with 300+ mods, you'll be faced with the hard decision of deactivating some mods. Luckily, dummy plugins you can just unpack the BSA and use the resources as Loose Files, then you don't need the plugin anymore. Other plugins can be merged via Wrye Bash into the Bashed Patch, further cutting down your mod count. And hopefully an equivalent of TES4Gecko will come out eventually, which will be able to merge mods into a single plugin. This last one however has an impact on your saved games, so it's for the more advanced user only.

 

Needs Investigation: Another small strangeness associated with Plugin BSAs, at least with pre-1.4.26 - they aren't loaded until you load either a saved game, or start a new game. That means if you're still at the Main Menu for the first time, any replacers loaded this way haven't taken effect yet! Woah! Not a big deal, unless you're trying to replace the Main Menu Logo... Oh wait. That mod works fine when installed as Loose Files, or if the BSA is registered in your Skyrim.ini, but it only works as a Plugin BSA after loading a saved game. This behavior still need to be verified now that 1.4.27 Skyrim is out.

Loose Files
This one's simple. If the file exists in the Data folder, it will always win. It will always be used, even if the same file exists in a Registered BSA or Plugin BSA. This is how most texture replacers work, and this is what mod managers like Wrye Bash or NMM are made for. Managing the Loose Files so you don't have to track them yourself. It's easy to keep track of which file came from which installer when you only have 3 mods. But when you get up to 150 mods, each with their own resources, a lot of them overlapping, a mod manager just takes the headache out of it.

2. What This Means For Mod Users
Not much really. There is one minor disadvantage to all this: Steam Workshop uses BSAs by default, even for texture replacers that don't need a plugin. That means you end up with an ESP that is only there to load a BSA, taking up one of your 253 mod slots doing it. If you start getting close to the 253 mod limit, you can cut back by either:

  • (Recommended) Unpacking those BSAs to Loose Files. A useful tool for unpacking the BSA is DDSopt. Once the BSA is unpacked, just pack up those Loose Files along with any documentation into an archive, and install with your favorite mod manager. Now you can deactivate the plugin, saving you a mod slot.
  • (Not Recommended) Register the BSAs. This method you'll use your Skyrim.ini to load the BSAs instead of using a plugin. First you'll need to deactivate or delete the plugin. Next, you'll need to add the name of the BSA to your Skyrim.ini, in the following section:
    [Archive]
    sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa
    sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa
    :  The reason I don't recommend this method is that it's harder to change the order. This method you need to edit your Skyrim.inieach time you want to change the order, whereas with Method #1, you just either change the Installer Order of the package in BAIN, or in NMM. Also, you'll run up on the 255 character limit pretty quick, which means that you'll need to make use of Nitpick, which in turn requires SKSE. There's nothing wrong with requiring SKSE, but you'll have to wait for a SKSE update after each game update.

Again, each of these tricks only works if the plugin in question is an empty, dummy plugin, solely there for the purpose of loading the BSA. You can check this using TES5Edit or another similar utility, or right click on the mod in Wrye Bash and select 'Mark Mergeable...' to find out if the plugin is empty. Wrye Bash will tell you if the plugin is empty, but it won't say anything at all relating to its "emptyness" if it's not empty.

3. Other Questions
Replacing Scripts
Pre-1.4.26 for Skyrim, you couln't replace a Vanilla Script file with one in a Plugin BSA. This is no longer true, it has been fixed, and script replacers should work as both Loose Files and Plugin BSAs now.

Directory Thrashing
What is it? link. Basically, simply having too many ESP, ESM, and BSA files (even inactive) in the data directory would cause Oblivion to go crazy, bringing your game to a crawl as disk I/O went through the roof. It's why Wrye Bash has the Auto-Ghost feature.

Does it affect Skyrim? Sort of, but not in the old sense of the term. There's a different problem with Skyrim, such that if you have over 508 mod files in your data directory, the engine just plain refuses to load any of them, active or not. To fix this, use Nitpick. Wrye Bash's ghosting will also work here since, Wrye Bash will not allow more than 255 active plugins at once, and ghost the rest, meaning you will at most have 255 files in the data directory with .esm or .esp extensions.

There's also the added problem of plugin INI files (which pretty much no one uses - no fix for this), and part of the old thrashing problem, where the game tries to read inactive plugins. For the last issue, again Wrye Bash's Ghosting feature still works good.

What are all these terms you use?
I used a bunch of terminology here, most of it old, but there's a couple new terms I made up for this explanation.

  • Plugin - any .esp or .esm file.
  • ESM - Elder Scrolls Master. Usually saved as a .esm file, which has the same format as a .esp file, only one bit (as in, one bit of a byte) is different.
  • ESP - Elder Scrolls Plugin. Most mods are ESPs, but some are ESMs. Their format is the same, but the game handles them slightly differently in some specific cases.
  • ESS - Elder Scrolls Save. Your saved games. This is loaded after all the ESPs and ESMs are loaded, so that its changes (like what items are in which containers) can override what the plugins say.
  • BSA - Bethesda Softworks Archive. Sort of like a .zip file, but specific to Bethsoft's games, and optimized (sort of) for their specific usage.
  • Load Order - The order that your mods (ESPs and ESMs) are loaded. When conflicts occur, the last loaded conflict "wins".
  • Install Order - The order that files are installed into your Data folder. When multiple packages install the same file, the last installed package "wins".
  • BSA Load Order - The order that the game engine loads BSAs. When conflicts in data files occur, the last loaded BSA "wins".
  • Plugin BSA - A BSA that the engine loads because an ESP/ESM with the same name is active.
  • Registered BSA - A BSA that the engine loads because it's listed in Skyrim.ini.
  • Loose File(s) - A resource file in the Data folder that's "loose", as in not packaged into a BSA.
  • FormID - A unique ID assigned to each "record" of a plugin. Each object in the game is defined by a record, and each record has a FormID. A FormID is 32-bits, with the first 8 bits defining which ESM or ESP the record belongs to.
  • SKSE - Skyrim Script Extender. A nice utility that everyone should be using. It aims to extend the functions available to scripts written for Skyrim. It's also useful as DLL loader, if one wants to inject some code into Skyrim's game engine.
  • SD - Script Dragon. Another DLL loader, but with a different goal in mind than SKSE. Both can work along side each other.
  • Vanilla - A term referring to Skyrim as distribute by Bethesda and Steam. This means no modifications have been done. Alludes to the "plainness" of the vanilla ice-cream flavor.

What about Nitpick?
I'm still maintaining it. shadeMe may also release updates as well, he has access to the Skyrim Nexus page, so either he or I can update it. The Nexus page also has links to the source code, but here's shadeMe's source and my source, links in case the Nexus is down at the moment.

4.Addendum: BSA Compression
This question comes up often enough to deserve an answer here: What's better, compressed BSAs or uncompressed BSAs?

To quote Ethatron:
 

Ethatron said

So lets see a little calculation for two operations of reading data from disk, uncompressed and then compressed:

  • Harddisk-throughput: 20MB/s
  • Memory-throughput: 2000MB/s
  • zlib-throughput: 80MB/s
  • uncompressed size: 100MB
  • compressed size: 60MB
This yields:
  • Reading 100MB at 20MB/s from disk takes 5s.
  • Reading 60MB at 20MB/s from disk takes 3s, decompressing 60MB at 80MB/s in-memory takes 0,75s, so a total of 3,75s.
In case the data is very small it's probable that seeking time will start dominating the times:
  • Reading 1MB at 20MB/s from disk takes 0,05s.
  • Reading 0,6MB at 20MB/s from disk takes 0.03s, decompressing 0,6MB at 80MB/s in-memory takes 0,0075s, so a total of 0,0375s.
Seeking may be as much as 0,002s. Nevertheless it doesn't change the fact that reading compressed data is faster as long as the decompressor is faster than the disk. It's a rather theoretic notion, but when using compressed solid archives even the seek-times go down as the disc-head has to reposition a smaller distance while bulk-reading from the same archive.

Also, in reality with a modern system the decompression speed of zlib reach realistic 250-500MB/s (as you can see here), which can be far beyond any SSD-speed.

 

So for the most part, compression is good for you. There are a few cases where this might be detrimental though: At very high compression rates, it seems some files are affected - notably sound files seems to take on a "fuzzy" sound to them in many cases. Textures and Meshes should be fine going to max compression, but be sure to test things if you're going to up the compression on your BSAs. 

 

 

Edit by Alt3rn1ty typo's plus .. : For reference as to what not to compress at High Compression levels - See this topic first post ( Any original BSA flagged as UnCompressed by Bethesda, is a pretty good bet contains 'some' files ( not necessarily all of them ) which should not be compressed )

Spoiler

 

Credit to Fireundubh on STEP forum for finding out the spoilered information



#2

    Mere Morsel

  • Account closed
  • PipPipPipPipPip
  • 3,566 posts

Users of older Mod Organiser ( MO ) versions take note of that quote from Ethatron at the end of the first post
 
There is a recurring opinion some users of MO have, in that they believe extracting all BSAs is a more optimal setup ..

Take two circumstances :

1. Original file compressed ( much smaller ) in a BSA on Hard Drive, or ..
2. Original file decompressed so that it is much larger on the same device

Decompressing a small file into a larger file in memory ( from where the file will then be used ), is executed much faster than reading a larger file into memory from your HD ( even if the HD is a SSD, the throughput is still much faster )
The reason is processors these days are much faster at decompressing a file than a hard drive is at reading a file, so the quicker loaded small file which has to be decompressed ends up being faster in execution than just reading a much larger file from the same device.

Reference the quote in "
4.Addendum: BSA Compression", end of first post above - Ethatron who created BSAOpt and its compression / decompression code did plenty of testing in this department to draw that conclusion, backed up with the math employed to calculate the results found.

The idea that having the BSAs all extracted for more optimum reading is flawed.

The game reads its files much quicker from compressed BSAs.
 

Additionally having all files loose, and therefore more fragmented across the device, makes seek times for all the individual blocks to recreate the file for reading much slower.

It also does not matter that MO uses a virtual device and does not load them from Skyrim data, it still has to load them from a different part of the hard drive where they are stored by MO.

 

The author of MO recognised how problematic extracting BSAs could be technically and so changed the default of allowing BSAs to be managed and extracted, to not allowing it, and separated the ability in a way that made users seek it out to enable it.

 

In other words its an advanced feature that you turn on at your own risk, and cannot blame mod authors for the complicated problems YOU create by turning the feature on.
 

See this topic - Ramifications of BSA Extraction in Mod Organiser

If you read nothing else, read the first post in its entirety, and Post #13 by the author of MO, in which Tannin not only explains why he stopped the ability being a default feature in more recent version of MO ....

 

... but also concurs that BSAs are more optimal in use than loose files. Read the part where Tannin mentions the Pros of using BSAs ..

 

Regarding BSA extraction in MO: I've decided to move this functionality into a plugin that will by default be disabled. This means an advanced user who wants BSA extraction can enabled it and use it as he always has whereas someone who doesn't know about BSA extraction will not be confronted with the choice at all.

Regarding BSA order vs Loose order in MO:

  • MO totally "messes up" this concept, just throw out everything you've read about BSA vs. loose order, it doesn't apply to MO.  In MO, (if you check all bsas in the archives tab) all BSAs are registered BUT if a BSA has higher installation order than a loose file that loose file doesn't appear in the VFS. This means assets are effectively loaded in installation order no matter which format they are in!
  • This also means that no matter if you extract BSAs or not, the bsa order will never be automatically matched to your plugin order!
  • Therefore you have to have to have to follow MOs suggested installation order in the warnings window (Potential mod order problems) to ensure a stable game.

When I suggested I might remove BSA extraction I was also thinking of removing the behaviour in 1, because then we could drop mod order suggestions as well -> loose one feature, get rid of one issue, drop tons of complicated and confusing functionality.
 
More BSA Pros:

 

  • Better performance ( a lot of files also slows down game startup, especially when using MO )
  • Less disk usage ( because BSAs can be compressed )
  • Faster installation ( you save yourself the extraction step )
  • Removes user-control ( yes, this is a good thing, at least for mod authors and myself wink.png )

 

 

 

I used to have a very weak machine which was only capable of running Skyrim if I employed a lot of tricks to make the game more optimal. I spent a couple of years making a mod which reduced as much of the games textures ( 32 thousand of them ) to more optimal sizes without losing too much detail, and have a lot of practical experience judging the effect of Loose files versus BSAs

 

After a great deal of testing I came to the same conclusion as Ethatron / Tannin / UPP Team and anyone else who has seriously taken time to study this, that the mod is best distributed as highly compressed BSAs for the Skyrim game, even if only for performance gains in my case.

 

Bethesda use BSAs for very good reasons : Which has become more evident recently as we discover the contents of Fallout 4 BSAs, Textures are notably now split ( which is why the Textures BSA is now a special case for extracting and compressing ), all the MipMaps are in a separate file, and the Large top layer image of the texture only loads as required by the game when the texture needs to be more detailled at close range - Otherwise only the MipMaps file loads into memory for speedy cached use at the more often distant ranges you will see the texture in game. Having all of Fallout 4's textures as loose files with a large top layer image with attached mipmap layers in the same file, is definitely less optimal than the vanilla FO4 game setup.

 

 

Update : The above will no longer be a problem in future versions of MO - Version 2 will no longer have the feature to manage extracting BSAs .. See a more recent quote from Tannin :

 

 

 The feature has been removed because it required a lot of complex game-specific and error-prone code inside the hooking lib and caused ton of confusion, bug reports and hatred from other modders.

It made the whole PMOP business necessary which way too many people didn't understand.

The bsa management was supposed to make things more intuitive by giving assets the "expected" order, but in practice it made everything more complicated.

With MO2 my primary focus is Fallout 4 where I don't want to repeat the same mistake and non-gamebryo games where it's unnecessary.

 

 

If you need this functionality for step then you should stick with MO1.

 

 

.



#3

    Mere Morsel

  • Account closed
  • PipPipPipPipPip
  • 3,566 posts

Using the official Archiver.exe

 

Arthmoor has a tutorial on using that along with Wrye Bash BAIN for putting your mods together :

 

 

Using BAIN and archive.exe to package a skyrim mod



#4

    Mere Morsel

  • Account closed
  • PipPipPipPipPip
  • 3,566 posts

Using BSAOpt

 

( At time of writing this post, I am using BSAOpt Beta 2.0.0 - Not the other one )

 

 

Just a few notes on the use of this tool, which has not had any documentation for a long while ( author is not active and the support site has been down for a long time )

 

 

Be sure you have selected "Skyrim" in the game menu before creating your BSA - If you dont, BSAOpt's default mode of Oblivion format BSAs will be used .. and they will crash Skyrim when you try to use them ( different games have different BSA formats as the needs for these archives evolve ).
 
 
If extracting BSAs - Choose source BSA at the top (browse to it)
Choose destination at the bottom (browse to a folder and click "Use folder" <- It will be extracted here)
Then click "Unpack"
 
If packing BSAs - Browse to the folder 'containing' your {textures} folder (ie dont go inside the textures folder) and click "Use Folder"
At the bottom browse to the folder where you want to create the BSA, give it a name.bsa, and choose "Save"
Make sure you select Skyrim in the game menu
Then click "pack"
 
 
133263-1466923744.jpg
 

 

 

Note that this tool can compress files to a greater degree than the official Archiver.exe from Bethesda ( when you install the Creation Kit )

 

Reference the spoiler at the end of the first post in this topic which details flags on file types, some original game BSAs are not compressed for good reason, because compressing some file types will cause loading and timing problems for the game engine ..

 

 

 

Edit by Alt3rn1ty typo's plus .. : For reference as to what not to compress at High Compression levels - See this topic first post ( Any original BSA flagged as UnCompressed by Bethesda, is a pretty good bet contains 'some' files ( not necessarily all of them ) which should not be compressed )

Spoiler

 

Credit to Fireundubh on STEP forum for finding out the spoilered information

 

 

 

So beware using BSAOpts higher compression levels on some files, notably the biggest problems are evident when sound files ( music / voice / effects ) are compressed further than they already are ( Sound files own format wav / mp3 / wma / xwma / xwm etc all have their own compression in the format of the filetype, adding further compression to the BSA they are stored in, means they effectively have to be decompressed twice before they are used in memory - Too much compression on the BSA aswell as having already a high level of xwm compression causes timing problems so some files dont play properly ).

 

But if you include just files such as textures and meshes .. you can go up to BSAOpts level 10 compression on its resulting BSAs without issue

 

 

If in doubt use the official Archiver.exe



#5

    Mere Morsel

  • Account closed
  • PipPipPipPipPip
  • 3,566 posts

Anyone wishing to copy / paste any useful info from the above posts, which may be relevant and useful in future game modding topics .. Feel free, and keep the community knowledge current and progressing.






Page loaded in: 1.061 seconds