[LE] [Update] BSAs and You

[Deleted133263User]

Due to Bethesda forums going to a different forum ( Bethesda.net ), reproducing an important topic here on Nexus just in case all old topics fall off the face of the internet as Bethesda move on, and the old locked forums are no longer maintained.

Original research and credit to Lojack and Wrinklyninja, I am just copying the info to preserve it on Nexus and AFK Mods

---------------------------------------------------------------------------

BSAs and You

Lojack : "We're back, with an update on the information provided in [iMPORTANT] BSAs and You!

First off, make sure you've read [iMPORTANT] Load Order and You, by wrinklyninja.

 

 

Load Order and You

Introduction

Load ordering is the method used to determine how conflicts between mod plugins (.esp, .esm files) should be decided. If two plugins alter the same game data, then the changes made by the plugin loading later will override those made by the plugin loading earlier. This "rule of one" results in a list of plugins, with those earlier in the list having any conflicting changes overriden by those later in the list. This list is the load order of the plugins.

A game will only load the plugins that are active. Up to 255 plugins, including the game's .esm file, can be active at any one time. Active plugins are listed in the game's "plugins.txt" file, which is stored in the user's local application data folder. Nevertheless, it is useful when working with load orders to consider the load order of all plugins, even if only some of them will actually be loaded. This is both because it is easier to display a single list of plugins than a list and an unordered set, and because modders have engineered methods that allow the changes made by inactive plugins to be loaded by another plugin (eg. Wrye Bash's Bashed Patch). When any such methods are being used, the load order of inactive plugins decides which plugins override others, similar to as if they were active.

In Oblivion, Nehrim, Fallout 3, Fallout: New Vegas and early versions (pre-1.4.26) of Skyrim, load order is decided by the relative timestamps of plugins in the game's Data directory. An installed plugin's load order is therefore an intrinsic property of that plugin.

In Skyrim v1.4.26, a new textfile-based load order system was introduced, in which load order is decide by the order in which plugins are listed in "plugins.txt". This brought with it a fundamental change, in that load order is no longer an intrinsic property of a plugin. This has the result that inactive plugins do not have any load order.


The Solution

The solution agreed on by Lojack (Wrye Bash), Kaburke (Nexus Mod Manager), WrinklyNinja (BOSS) and Dark0ne (owner of the Nexus sites) was that total load order would be stored in a "loadorder.txt" file, itself stored in the same location as "plugins.txt". "plugins.txt" would be kept in synchronisation with "loadorder.txt" so that the order of plugins that the game loaded was the same for both files, but the latter would allow the load ordering of inactive plugins.

Modding utilities would then perform their changes on "loadorder.txt", updating "plugins.txt" to reflect any changes to active plugin load order as required. This provides a common store for the total load order in lieu of the plugin timestamps used by the other games. There are of course a few more technical details to it, but that's the basics.


What This Means For... Mod Makers & Users

The bad news: Any utilities that you use to manage load order may stop having any effect. This is because nothing currently released knows about the change in load order system. In time, utilities may be updated to handle the text-based system as their programmers become aware of it. Until then, Skyrim's launcher is the only way to change load order. Also be aware of the current limitations of the solution, given above.

One very important thing is that if you have a utility that can handle the text-based system, do not use Skyrim's launcher to set load order. This is because it doesn't know about the total load order, so if you change the load order through it, only the active load order changes and synchronisation between the two is lost. The only way for a utility to re-sync the two load orders is to undo the changes you made in the launcher.

The good news: Nothing changes when it comes to making or using mods, it's just business as usual. Just bear in mind the point about the utilities you use to manage load order needing updating, and have patience while the programmers do their thing.

If you want to manually set your load order, the instructions below may be of use:

• Open your plugins.txt in a text editor. plugins.txt is found in your local application data folder. On Windows Vista/7, this is C:\Users\[username]\AppData\Local\Skyrim\plugins.txt.
• In plugins.txt, list the plugins you want active in the load order that you want them in. Each plugin goes on a separate line. Master files should go before non-master files. Merged or imported plugins are not active plugins, so should not be listed in plugins.txt.
• If you also have a loadorder.txt in the same folder as plugins.txt, then update it so that the plugins in plugins.txt are in the same order in both files.
• Save the edited file(s) and quit.
• Don't try to then edit your load order through the Skyrim launcher. It has been reported to reset your load order if you set it up manually.

What This Means For... Mod Utility Programmers

The bad news: Your utilities will no longer function when it comes to load order (both getting and setting) until you update them to support the current system.

The good news: I've already done a lot of the work for you. The BOSS API supports both the timestamp-based and text-based load order systems through the same interface, so you don't need to worry about the differences, and includes all the functions required for querying and changing load order and activation status. The BOSS API will be included in the v2.0 release of BOSS, but if you PM me I'll send you a copy if v2.0 hasn't been released yet.

Note that the BOSS API is licensed under the GNU GPL v3.0 license, so your utility's license will have to comply with that to use the API.

If your utility's license is such that you cannot use the API, it's still worth looking at the BOSS API's readme as that contains the information on the textfile-based load order standard. If BOSS v2.0 hasn't been released yet, PM me and I'll send you a copy of the readme.



Final Notes

Please remember to have patience if you're waiting on someone to do something in relation to this issue, eg. users waiting on programmers, or programmers waiting on me to fix an API bug. Everyone who can do anything about this situation is pretty busy, and these things take time to get done right.

Also please don't hate on Bethesda for changing the system. It's better than the old one, and the issues that it brought up are because we're too smart for our own good, coming up with ways to bend the rules (eg. Bashed Patches), so it's wrong to think that Bethesda need to cater for the situations in which they arise.

It's probably a good idea if people post below any utilities that use load order in some way that aren't listed above, so that I can add them to the compatibility list.

Finally, spread the word. If someone's having trouble setting the load order, let them know that it might be to do with this. If you know of a modding utility not listed above that changes or uses load order in any way, let the author(s) know about this. Please be polite while doing so though, I don't want to end up reading verbal abuse resulting from this instruction. When letting utility authors know, don't demand they update their utility, just post a note saying that it doesn't work any more and link to this thread.

Related links:
Thread #1 on the BethSoft forums
Thread #2 on the BethSoft forums
This thread on the Nexus forums
News item for the issue on Skyrim Nexus

 

 

 

With the latest Beta patch for Skyrim officially getting sent out to all clients, there have been some changes to BSA Load Ordering that you might want to know about. Basically, the issues brought up in the previous thread have been fixed. So this post is only here for educational purposes, in case you're curious as to the order of resources loading.

To summarize, the order resources are loaded is:

Registered BSAs -> Plugin BSAs -> Loose Files

With Registered BSAs being loaded in the order listed in Skyrim.ini, and Plugin BSAs loading in the same order as their plugins, which are loaded in the order they are listed in plugins.txt.

Disclaimer: The game doesn't actually work exactly like this. This is how it logically works. As in, if you had to describe the decision process the game goes through, this is it. Realistically, working this way would be a waste of resources and processing time (why load up every file, when you already know a different one is going to "win"?). It's just much easier to understand if you think of it like this. A good description of how it would actually work is here.

 

Post from JDFan :

LoJack --- Is this correct or does it actually SEARCH for the textures in the opposite order UNTIL it finds the texture it is looking for ?? ( seems it would make more sense for the game to go through the list backwards until it finds the texture it is looking for and then load the one it finds first instead of searching entirely through every file each time !! )

Also IIRC it also searches RAM first and if the texture already exists in RAM it does not even search the files ( this is why it is important to exit the game when testing so that changes will show up properly otherwise RAM may not update and you'll get different results than when actually playing -- at least this was the case with OB so if you had 2 mods that each used a different version of a texture with the same name sometimes you'd get the vcorrect texture and other times you'd get a different one if the second model had recently loaded and was stil stored in RAM.

While the end result is the same in most cases whether it actually loads every file first and then overwrites again as it finds newer ones it seems that would cause a lot of problems and slowdowns vs. traversing the lists backwards until it finds the first instance of the texture (which would be faster and consume less resources !

 

 

 

 

Contents:

• Resource Load Order
  • Registered BSAs
  • Plugin BSAs
  • Loose Files
• What This Means For Mod Users
• Other Questions
  • Replacing Scripts
  • Directory Thrashing
  • What are all these terms you use?
  • What about Nitpick?
• Addendum: BSA Compression

1. Resource Load Order

Skyrim loads its resources from the Data folder. Makes sense. These resources can also be packed up into a nice neat little archive, called a BSA. These are Bethsoft's custom archive files, and internally they mirror the layout of the Data folder. They're nice, because they keep the Data folder uncluttered, and it makes it really easy to remove every file from one mod.

Ok, but what happens when the same resource exists in more than one place? For example textures/dungeons/imperial/battlemap01.dds, which is the texture for the map you see all the time in castles, with the little flags and stuff sticking out of them. This file exists in Skyrim - Textures.bsa. That's the original file. If you download a texture replacer, you'll get it as a loose file probably, and if you download HD Skyrimmap from Steam Workshop, you'll get that file in a BSA. So which file actually gets used? Well, which ever one is loaded last. The game loads resources in three stages, with the last files loaded being what you see in game:

Needs Investigation: With pre-1.4.26 Skyrim, only Registered BSAs and Loose Files were loaded when the game first loads. That means anything in a Plugin BSA won't take effect yet - but you'll only notice that if you're trying to replace a Main Menu asset, like the logo. Once you start a game, either by loading a saved game or starting a new one, then the Plugin BSAs are loaded. So in the example of Brumbek's Main Menu Logo replacer, if it's loaded via a Plugin BSA (like it is when installed via Steam Workshop), you won't see the spinning logo until you load a game, then exit back to the Main Menu.

So the investigation needed: Is this still the case? I'll get back to you on that once I get the time to test it.

Registered BSAs
First the game reads your Skyrim.ini file. There's a section that tells the game which BSAs to load. It looks like this for most people:

[Archive]
sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa
sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa

The game will load the BSAs in order as listed, BSAs in sResourceArchiveList first, and sResourceArchiveList2 second. So if you manually register a texture replacer BSA (like the HD Texture packs), but put it at the beginning of the list - they won't load! Because the Skyrim - Textures.bsa will override them of course. If you put it after, or at the end of the list, or in the second entry, then they will load.

Easy enough, but there is one drawback: There's a 255 character limit on each of these entries. This is because the game internally uses a 256 byte character buffer to receive the value when reading it. One byte is reserved for the NULL terminator (a byte with a value of 0, indicating the end of the data), so that leaves 255 characters for file names. If you want to get around this limitation, you'll need to use Nitpick.

Plugin BSAs
Next the game loads any BSAs associated with a plugin. When the game loads an active ESP, it also looks for a BSA of the same name (For Oblivion, there were naming conventions that allowed a single ESP to load multiple BSAs in this manner - this is not the case for Skyrim). So dummy.esp will make the game load dummy.bsa. This is true even if the ESP does absolutely nothing (for example, the official HD Texture packs do this, as well as Oblivion's Shivering Isles expansion). Cool! We found an easy way to make our replacement textures only load when we want! Just deactivate the plugin and they don't load, activate it and they do!

Ok, but what happens if two Plugin-loaded BSAs have the same file in it? Well, it's easy now: whichever one loads last wins (as usual). But now there's a reliable way to control which order they're loaded in: they're loaded in the same order as their associated plugins. This means if dummy.esp loads before another.esp, then dummy.bsa will also load before another.bsa, meaning of course that another.bsawill win, because it loaded last.

Sounds good, but I seem to remember there being other issues with using BSAs...

Yes, nothing major really, unless you're trying to mod your game with a lot of plugins.

 

Due to the way the game indexes plugins and stores the data in your save game, you can only have 256 active mod files. That's because objects are referenced by what's called a FormID, which is a 32-bit (4-byte) unique number, usually written as an 8-digit hexadecimal number. The first two hex digits are used to identify which mod the data came from. Two hex digits can represent a maximum of 256 unique numbers (0-255).

 

Now, Skyrim.esm and Update.esm will always fill up two of those. Another one that's not obvious is your saved game itself - the one in use is always assigned the ID of hexadecimal FF, or 255. That leaves room for 253 other plugins.

 

And here we get to our point: dummy plugins that don't do anything at all, but exist solely to make the game load a BSA? Yep, those take up one of our plugin slots. This might not seem like a big deal, but once you start trying to play with 300+ mods, you'll be faced with the hard decision of deactivating some mods. Luckily, dummy plugins you can just unpack the BSA and use the resources as Loose Files, then you don't need the plugin anymore. Other plugins can be merged via Wrye Bash into the Bashed Patch, further cutting down your mod count. And hopefully an equivalent of TES4Gecko will come out eventually, which will be able to merge mods into a single plugin. This last one however has an impact on your saved games, so it's for the more advanced user only.

 

Needs Investigation: Another small strangeness associated with Plugin BSAs, at least with pre-1.4.26 - they aren't loaded until you load either a saved game, or start a new game. That means if you're still at the Main Menu for the first time, any replacers loaded this way haven't taken effect yet! Woah! Not a big deal, unless you're trying to replace the Main Menu Logo... Oh wait. That mod works fine when installed as Loose Files, or if the BSA is registered in your Skyrim.ini, but it only works as a Plugin BSA after loading a saved game. This behavior still need to be verified now that 1.4.27 Skyrim is out.

Loose Files
This one's simple. If the file exists in the Data folder, it will always win. It will always be used, even if the same file exists in a Registered BSA or Plugin BSA. This is how most texture replacers work, and this is what mod managers like Wrye Bash or NMM are made for. Managing the Loose Files so you don't have to track them yourself. It's easy to keep track of which file came from which installer when you only have 3 mods. But when you get up to 150 mods, each with their own resources, a lot of them overlapping, a mod manager just takes the headache out of it.

2. What This Means For Mod Users
Not much really. There is one minor disadvantage to all this: Steam Workshop uses BSAs by default, even for texture replacers that don't need a plugin. That means you end up with an ESP that is only there to load a BSA, taking up one of your 253 mod slots doing it. If you start getting close to the 253 mod limit, you can cut back by either:

• (Recommended) Unpacking those BSAs to Loose Files. A useful tool for unpacking the BSA is DDSopt. Once the BSA is unpacked, just pack up those Loose Files along with any documentation into an archive, and install with your favorite mod manager. Now you can deactivate the plugin, saving you a mod slot.
• (Not Recommended) Register the BSAs. This method you'll use your Skyrim.ini to load the BSAs instead of using a plugin. First you'll need to deactivate or delete the plugin. Next, you'll need to add the name of the BSA to your Skyrim.ini, in the following section:
  

  [Archive]
  sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa
  sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa

  : The reason I don't recommend this method is that it's harder to change the order. This method you need to edit your Skyrim.inieach time you want to change the order, whereas with Method #1, you just either change the Installer Order of the package in BAIN, or in NMM. Also, you'll run up on the 255 character limit pretty quick, which means that you'll need to make use of Nitpick, which in turn requires SKSE. There's nothing wrong with requiring SKSE, but you'll have to wait for a SKSE update after each game update.

Again, each of these tricks only works if the plugin in question is an empty, dummy plugin, solely there for the purpose of loading the BSA. You can check this using TES5Edit or another similar utility, or right click on the mod in Wrye Bash and select 'Mark Mergeable...' to find out if the plugin is empty. Wrye Bash will tell you if the plugin is empty, but it won't say anything at all relating to its "emptyness" if it's not empty.

3. Other Questions
Replacing Scripts
Pre-1.4.26 for Skyrim, you couln't replace a Vanilla Script file with one in a Plugin BSA. This is no longer true, it has been fixed, and script replacers should work as both Loose Files and Plugin BSAs now.

Directory Thrashing
What is it? link. Basically, simply having too many ESP, ESM, and BSA files (even inactive) in the data directory would cause Oblivion to go crazy, bringing your game to a crawl as disk I/O went through the roof. It's why Wrye Bash has the Auto-Ghost feature.

Does it affect Skyrim? Sort of, but not in the old sense of the term. There's a different problem with Skyrim, such that if you have over 508 mod files in your data directory, the engine just plain refuses to load any of them, active or not. To fix this, use Nitpick. Wrye Bash's ghosting will also work here since, Wrye Bash will not allow more than 255 active plugins at once, and ghost the rest, meaning you will at most have 255 files in the data directory with .esm or .esp extensions.

There's also the added problem of plugin INI files (which pretty much no one uses - no fix for this), and part of the old thrashing problem, where the game tries to read inactive plugins. For the last issue, again Wrye Bash's Ghosting feature still works good.

What are all these terms you use?
I used a bunch of terminology here, most of it old, but there's a couple new terms I made up for this explanation.

• Plugin - any .esp or .esm file.
• ESM - Elder Scrolls Master. Usually saved as a .esm file, which has the same format as a .esp file, only one bit (as in, one bit of a byte) is different.
• ESP - Elder Scrolls Plugin. Most mods are ESPs, but some are ESMs. Their format is the same, but the game handles them slightly differently in some specific cases.
• ESS - Elder Scrolls Save. Your saved games. This is loaded after all the ESPs and ESMs are loaded, so that its changes (like what items are in which containers) can override what the plugins say.
• BSA - Bethesda Softworks Archive. Sort of like a .zip file, but specific to Bethsoft's games, and optimized (sort of) for their specific usage.
• Load Order - The order that your mods (ESPs and ESMs) are loaded. When conflicts occur, the last loaded conflict "wins".
• Install Order - The order that files are installed into your Data folder. When multiple packages install the same file, the last installed package "wins".
• BSA Load Order - The order that the game engine loads BSAs. When conflicts in data files occur, the last loaded BSA "wins".
• Plugin BSA - A BSA that the engine loads because an ESP/ESM with the same name is active.
• Registered BSA - A BSA that the engine loads because it's listed in Skyrim.ini.
• Loose File(s) - A resource file in the Data folder that's "loose", as in not packaged into a BSA.
• FormID - A unique ID assigned to each "record" of a plugin. Each object in the game is defined by a record, and each record has a FormID. A FormID is 32-bits, with the first 8 bits defining which ESM or ESP the record belongs to.
• SKSE - Skyrim Script Extender. A nice utility that everyone should be using. It aims to extend the functions available to scripts written for Skyrim. It's also useful as DLL loader, if one wants to inject some code into Skyrim's game engine.
• SD - Script Dragon. Another DLL loader, but with a different goal in mind than SKSE. Both can work along side each other.
• Vanilla - A term referring to Skyrim as distribute by Bethesda and Steam. This means no modifications have been done. Alludes to the "plainness" of the vanilla ice-cream flavor.

What about Nitpick?
I'm still maintaining it. shadeMe may also release updates as well, he has access to the Skyrim Nexus page, so either he or I can update it. The Nexus page also has links to the source code, but here's shadeMe's source and my source, links in case the Nexus is down at the moment.

4.Addendum: BSA Compression
This question comes up often enough to deserve an answer here: What's better, compressed BSAs or uncompressed BSAs?

To quote Ethatron:

Ethatron said

So lets see a little calculation for two operations of reading data from disk, uncompressed and then compressed:

• Harddisk-throughput: 20MB/s
• Memory-throughput: 2000MB/s
• zlib-throughput: 80MB/s
• uncompressed size: 100MB
• compressed size: 60MB

This yields:

• Reading 100MB at 20MB/s from disk takes 5s.
• Reading 60MB at 20MB/s from disk takes 3s, decompressing 60MB at 80MB/s in-memory takes 0,75s, so a total of 3,75s.

In case the data is very small it's probable that seeking time will start dominating the times:

• Reading 1MB at 20MB/s from disk takes 0,05s.
• Reading 0,6MB at 20MB/s from disk takes 0.03s, decompressing 0,6MB at 80MB/s in-memory takes 0,0075s, so a total of 0,0375s.

Seeking may be as much as 0,002s. Nevertheless it doesn't change the fact that reading compressed data is faster as long as the decompressor is faster than the disk. It's a rather theoretic notion, but when using compressed solid archives even the seek-times go down as the disc-head has to reposition a smaller distance while bulk-reading from the same archive.

Also, in reality with a modern system the decompression speed of zlib reach realistic 250-500MB/s (as you can see here), which can be far beyond any SSD-speed.

 

So for the most part, compression is good for you. There are a few cases where this might be detrimental though: At very high compression rates, it seems some files are affected - notably sound files seems to take on a "fuzzy" sound to them in many cases. Textures and Meshes should be fine going to max compression, but be sure to test things if you're going to up the compression on your BSAs.

 

 

Edit by Alt3rn1ty typo's plus .. : For reference as to what not to compress at High Compression levels - See this topic first post ( Any original BSA flagged as UnCompressed by Bethesda, is a pretty good bet contains 'some' files ( not necessarily all of them ) which should not be compressed )

 

 

I used my powers of awesomeness to determine the archive flags that were used to pack the official BSAs. This information could be important!

Skyrim - Animations.bsa
Flags: Compressed

Skyrim - Interface.bsa
Uncompressed, Flags: None

Skyrim - Meshes.bsa
Flags: Compressed, Retain Strings During Startup
Resource Types: Meshes

Skyrim - Misc.bsa
Uncompressed, Flags: Retain File Names

Skyrim - Shaders.bsa
Uncompressed, Flags: None

Skyrim - Sounds.bsa
Uncompressed, Flags: Retain File Names
Resource Types: Sounds, Voices

Skyrim - Textures.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

Skyrim - Voices.bsa
Uncompressed, Flags: None
Resource Types: Sounds, Voices

Skyrim - VoicesExtra.bsa
Uncompressed, Flags: None
Resource Types: Sounds, Voices

Dawnguard.bsa
Uncompressed, Flags: Retain File Names, Retain Strings During Startup
Resource Types: Meshes, Sounds, Textures, Voices

HearthFires.bsa
Uncompressed, Flags: Retain File Names, Retain Strings During Startup
Resource Types: Meshes, Sounds, Textures, Voices

Dragonborn.bsa
Uncompressed, Flags: Retain File Names, Retain Strings During Startup
Resource Types: Meshes, Sounds, Textures, Voices

HighResTexturePack01.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

HighResTexturePack02.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

HighResTexturePack03.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

Note:
By default, all BSAs have these flags: Include Directory Names and Include File Names. You can't change those flags, so I didn't add them to the list.

 

 

 

Credit to Fireundubh on STEP forum for finding out the spoilered information

[Deleted133263User]

Users of older Mod Organiser ( MO ) versions take note of that quote from Ethatron at the end of the first post

There is a recurring opinion some users of MO have, in that they believe extracting all BSAs is a more optimal setup ..

Take two circumstances :

1. Original file compressed ( much smaller ) in a BSA on Hard Drive, or ..
2. Original file decompressed so that it is much larger on the same device

Decompressing a small file into a larger file in memory ( from where the file will then be used ), is executed much faster than reading a larger file into memory from your HD ( even if the HD is a SSD, the throughput is still much faster )
The reason is processors these days are much faster at decompressing a file than a hard drive is at reading a file, so the quicker loaded small file which has to be decompressed ends up being faster in execution than just reading a much larger file from the same device.

Reference the quote in "4.Addendum: BSA Compression", end of first post above - Ethatron who created BSAOpt and its compression / decompression code did plenty of testing in this department to draw that conclusion, backed up with the math employed to calculate the results found.

The idea that having the BSAs all extracted for more optimum reading is flawed.

The game reads its files much quicker from compressed BSAs.

Additionally having all files loose, and therefore more fragmented across the device, makes seek times for all the individual blocks to recreate the file for reading much slower.

It also does not matter that MO uses a virtual device and does not load them from Skyrim data, it still has to load them from a different part of the hard drive where they are stored by MO.

 

The author of MO recognised how problematic extracting BSAs could be technically and so changed the default of allowing BSAs to be managed and extracted, to not allowing it, and separated the ability in a way that made users seek it out to enable it.

 

In other words its an advanced feature that you turn on at your own risk, and cannot blame mod authors for the complicated problems YOU create by turning the feature on.

See this topic - Ramifications of BSA Extraction in Mod Organiser

If you read nothing else, read the first post in its entirety, and Post #13 by the author of MO, in which Tannin not only explains why he stopped the ability being a default feature in more recent version of MO ....

 

... but also concurs that BSAs are more optimal in use than loose files. Read the part where Tannin mentions the Pros of using BSAs ..

 

Regarding BSA extraction in MO: I've decided to move this functionality into a plugin that will by default be disabled. This means an advanced user who wants BSA extraction can enabled it and use it as he always has whereas someone who doesn't know about BSA extraction will not be confronted with the choice at all.

Regarding BSA order vs Loose order in MO:

• MO totally "messes up" this concept, just throw out everything you've read about BSA vs. loose order, it doesn't apply to MO. In MO, (if you check all bsas in the archives tab) all BSAs are registered BUT if a BSA has higher installation order than a loose file that loose file doesn't appear in the VFS. This means assets are effectively loaded in installation order no matter which format they are in!
• This also means that no matter if you extract BSAs or not, the bsa order will never be automatically matched to your plugin order!
• Therefore you have to have to have to follow MOs suggested installation order in the warnings window (Potential mod order problems) to ensure a stable game.

When I suggested I might remove BSA extraction I was also thinking of removing the behaviour in 1, because then we could drop mod order suggestions as well -> loose one feature, get rid of one issue, drop tons of complicated and confusing functionality.

More BSA Pros:

 

• Better performance ( a lot of files also slows down game startup, especially when using MO )
• Less disk usage ( because BSAs can be compressed )
• Faster installation ( you save yourself the extraction step )
• Removes user-control ( yes, this is a good thing, at least for mod authors and myself http://afkmods.iguanadons.net/public/style_emoticons/default/wink.png )

 

 

 

I used to have a very weak machine which was only capable of running Skyrim if I employed a lot of tricks to make the game more optimal. I spent a couple of years making a mod which reduced as much of the games textures ( 32 thousand of them ) to more optimal sizes without losing too much detail, and have a lot of practical experience judging the effect of Loose files versus BSAs

 

After a great deal of testing I came to the same conclusion as Ethatron / Tannin / UPP Team and anyone else who has seriously taken time to study this, that the mod is best distributed as highly compressed BSAs for the Skyrim game, even if only for performance gains in my case.

 

Bethesda use BSAs for very good reasons : Which has become more evident recently as we discover the contents of Fallout 4 BSAs, Textures are notably now split ( which is why the Textures BSA is now a special case for extracting and compressing ), all the MipMaps are in a separate file, and the Large top layer image of the texture only loads as required by the game when the texture needs to be more detailled at close range - Otherwise only the MipMaps file loads into memory for speedy cached use at the more often distant ranges you will see the texture in game. Having all of Fallout 4's textures as loose files with a large top layer image with attached mipmap layers in the same file, is definitely less optimal than the vanilla FO4 game setup.

 

 

Update : The above will no longer be a problem in future versions of MO - Version 2 will no longer have the feature to manage extracting BSAs .. See a more recent quote from Tannin :

 

 

The feature has been removed because it required a lot of complex game-specific and error-prone code inside the hooking lib and caused ton of confusion, bug reports and hatred from other modders.

It made the whole PMOP business necessary which way too many people didn't understand.

The bsa management was supposed to make things more intuitive by giving assets the "expected" order, but in practice it made everything more complicated.

With MO2 my primary focus is Fallout 4 where I don't want to repeat the same mistake and non-gamebryo games where it's unnecessary.

 

If you need this functionality for step then you should stick with MO1.

.

[Deleted133263User]

Using the official Archiver.exe

Arthmoor has a tutorial on using that along with Wrye Bash BAIN for putting your mods together :

 

Using BAIN and archive.exe to package a skyrim mod

[Deleted133263User]

Using BSAOpt

 

( At time of writing this post, I am using BSAOpt Beta 2.0.0 - Not the other one )

 

 

Just a few notes on the use of this tool, which has not had any documentation for a long while ( author is not active and the support site has been down for a long time )

 

 

Be sure you have selected "Skyrim" in the game menu before creating your BSA - If you dont, BSAOpt's default mode of Oblivion format BSAs will be used .. and they will crash Skyrim when you try to use them ( different games have different BSA formats as the needs for these archives evolve ).

If extracting BSAs - Choose source BSA at the top (browse to it)

Choose destination at the bottom (browse to a folder and click "Use folder" <- It will be extracted here)

Then click "Unpack"

If packing BSAs - Browse to the folder 'containing' your {textures} folder (ie dont go inside the textures folder) and click "Use Folder"

At the bottom browse to the folder where you want to create the BSA, give it a name.bsa, and choose "Save"

Make sure you select Skyrim in the game menu

Then click "pack"

 

 

Note that this tool can compress files to a greater degree than the official Archiver.exe from Bethesda ( when you install the Creation Kit )

 

Reference the spoiler at the end of the first post in this topic which details flags on file types, some original game BSAs are not compressed for good reason, because compressing some file types will cause loading and timing problems for the game engine ..

 

 

Edit by Alt3rn1ty typo's plus .. : For reference as to what not to compress at High Compression levels - See this topic first post ( Any original BSA flagged as UnCompressed by Bethesda, is a pretty good bet contains 'some' files ( not necessarily all of them ) which should not be compressed )

 

 

I used my powers of awesomeness to determine the archive flags that were used to pack the official BSAs. This information could be important!

Skyrim - Animations.bsa
Flags: Compressed

Skyrim - Interface.bsa
Uncompressed, Flags: None

Skyrim - Meshes.bsa
Flags: Compressed, Retain Strings During Startup
Resource Types: Meshes

Skyrim - Misc.bsa
Uncompressed, Flags: Retain File Names

Skyrim - Shaders.bsa
Uncompressed, Flags: None

Skyrim - Sounds.bsa
Uncompressed, Flags: Retain File Names
Resource Types: Sounds, Voices

Skyrim - Textures.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

Skyrim - Voices.bsa
Uncompressed, Flags: None
Resource Types: Sounds, Voices

Skyrim - VoicesExtra.bsa
Uncompressed, Flags: None
Resource Types: Sounds, Voices

Dawnguard.bsa
Uncompressed, Flags: Retain File Names, Retain Strings During Startup
Resource Types: Meshes, Sounds, Textures, Voices

HearthFires.bsa
Uncompressed, Flags: Retain File Names, Retain Strings During Startup
Resource Types: Meshes, Sounds, Textures, Voices

Dragonborn.bsa
Uncompressed, Flags: Retain File Names, Retain Strings During Startup
Resource Types: Meshes, Sounds, Textures, Voices

HighResTexturePack01.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

HighResTexturePack02.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

HighResTexturePack03.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

Note:
By default, all BSAs have these flags: Include Directory Names and Include File Names. You can't change those flags, so I didn't add them to the list.

 

 

 

Credit to Fireundubh on STEP forum for finding out the spoilered information

 

 

 

So beware using BSAOpts higher compression levels on some files, notably the biggest problems are evident when sound files ( music / voice / effects ) are compressed further than they already are ( Sound files own format wav / mp3 / wma / xwma / xwm etc all have their own compression in the format of the filetype, adding further compression to the BSA they are stored in, means they effectively have to be decompressed twice before they are used in memory - Too much compression on the BSA aswell as having already a high level of xwm compression causes timing problems so some files dont play properly ).

 

But if you include just files such as textures and meshes .. you can go up to BSAOpts level 10 compression on its resulting BSAs without issue

 

 

If in doubt use the official Archiver.exe

[Deleted133263User]

Anyone wishing to copy / paste any useful info from the above posts, which may be relevant and useful in future game modding topics .. Feel free, and keep the community knowledge current and progressing.

[Dreifels]

A new manual, in which the information arising in the course of the development of CAO 64 is processed, which makes much of what was written earlier on the subject in need of revision, is available HERE as a 23 page PDF.

[BillyG]

From the help text of bsarch.exe: "Keep in mind that sounds and voices don't work in compressed archives in all Bethesda games! Even if your archive contains a single sound/voice file out of thousands, it must be uncompressed." ...

 

... what's "all" games, do compressed voice files work in Skyrim SE and Fallout 4?

[PapaiJarda]

From the help text of bsarch.exe: "Keep in mind that sounds and voices don't work in compressed archives in all Bethesda games! Even if your archive contains a single sound/voice file out of thousands, it must be uncompressed." ...

 

... what's "all" games, do compressed voice files work in Skyrim SE and Fallout 4?

I think he was referring to pre-Oldrim games.