I'm posting here as this issue could use some more exposure and I fear the timing of it's discovery with the release of Fallout 4 will mean it will go unaddressed if left much longer. I see it as a pretty serious threat to the longevity of Skyrim in general. It's a bit of a long, technical read but please try to bear with me and take the time to digest the post fully as there is a lot of information. I feel there are a lot more people out there have this issue and don't realize it because your first instinct would be that it's a mod conflict of some sort.
A limitation in Skyrims save file format has been discovered which causes saves to become unloadable - CTD on loading these affected saves. This is NOT random CTD on load that Skyrim does every now and then. These saves will never load.
This issue has been reported on:
STEP forum: http://forum.step-pr...the-save-files/
Bethesda forum: http://forums.bethso...-extender-skse/
and the SKSE team have been emailed about it.
Am I Affected?
To test whether or not you have this issue.
1. Download the Save game script cleaner by Hadoram: http://www.nexusmods...im/mods/52363/?
2. Open the save giving you trouble.
3. If the script cleaner immediately crashes it's likely this particular save has this issue.
4. If it does not crash. Happy days. You are not yet affected by this issue. Please try searching other threads regarding CTDs on load as this particular flavour of 'CTD on load' is very specific
How Close Are My Saves To Imploding?
1. Download TESV ESS Files Editor by fubrus: http://www.nexusmods...im/mods/27119/?
2. Open your suspect save.
3. Expand "Global data table 3"
4. Click on Papyrus (don't need to expand).
5. If you receive a pop up error "Range check error" your save is almost definitely screwed.
6. Note the "Strings" entry.
7. 65,535 - (the number of strings from step 6) = the cushion you have from having corrupt saves.
Technicial Issue Description:
Please see the following link: http://www.uesp.net/..._Format/Papyrus
Note the second entry
strCount is a 16 bit field. It's maximum value is FFFF hex or 65,535 decimal. Once the number of strings is incremented past 0xFFFF we're guessing the pointer 'wraps around' and resets to 0x0000 making the save unreadable. I've seen it happen where save 1 is corrupt and save 2 (5 mins later) is fine. It would seem to support the theory that the pointer is being incremented past 0xFFFF and then decremented again below 0xFFFF as the game is played and strings are created and discarded at the very limit of strCount. Corruption this way can occur in game while you are oblivious you are creating bad saves. IE. the game does not CTD nor is there any indication of any weirdness within papyrus while playing so it seems that the game engine itself is capable of strCount > 0xFFFF but the save file itself has become unfit for purpose.
Just to put it in context: Vanilla Skyrim + Dawnguard + Hearthfires + Dragonborn + Unofficial Legendary Patch + SkyUI + Live Another Life = A total of 28,619 strings. Nearly half of the field (43%) is consumed with an almost vanilla setup and that's a fresh game, sitting in the LAL cell. As far as we can figure, as more and more scripts are instanced as the game is played, more strings are added and strCount grows.
I would love to have a figure for strCount for this vanilla setup after say 100 hours of play but can not afford the time right now.
Something to bear in mind is that this is not a performance issue. It is simply a capacity issue. Some of the heaviest mods, string-wise (through no fault of their own), are not performance intensive at all. For example and I'm only using this mod as an example as people will be familiar with it and probably surprised with it's impact. Interesting NPCs consumes 3,471 strings (over 5% for a single mod) in a new game, again, just sitting in the LAL cell.
What Are Strings?
They appear to be:
- variable definitions for functions using local variables. Usually they are removed from the string table once a function has completed but some of them are persistent - MCM variables.
- script properties. The vast majority of the string table look like properties.
- script/fragment names minus the .pex file extension. These script names actually appear to be properties as well. It explains why there are very few script names appearing in the string table.
What Type Of Mods Have The Biggest Impact On strCount?
Mods with a large number of scripts. This is not necessarily true. But as a general rule of thumb, more scripts = more properties. It seems this is why large dialog mods (Interesting NPCs) have such a large impact. I'm no expert on modding dialogue but apparently scripts are required to simply advance quests stages. To me, it seems like a very, very DUMB method implemented by Bethesda for mod authors to add DIALOG to a RPG and then slap a 16 bit limit on it.
- Mods with expansive complex MCMs. Each MCM entry (slider, toggle, whatever) requires several variable definitions. Scrolling through the strCount table I see lots of variables with 'OID' in their name and they're just the variables from mod authors that use that convention. I sigh when I see mods adding pointless MCMs. Keep it simple folks.
- Scripts with lots of properties. Properties seem to be the biggest contributor to the size of the string table. For Eg. I recently 'down-graded' the excellent "Illustrious Cloaks Of Skyrim" from the scripted version to the crafting version. Scripted version has around 230 properties in one script. While the scripted version is a much more elegant and proper implementation, the crafting version has no scripts at all.
- Animation mods. Especially problematic for users on 'that other site' as each animation has been reported to add several strings.
Impact Of Mod Uninstallation:
Vanilla setup from above. strCount = 28,619.
Install interesting NPCs. strCount = 32,090.
Deactivate Interesting NPCs esp. strCount = 32,090.
Deactivate Interesting NPCs mod entirely (ie. remove scripts). strCount = 29,471.
Run the save through the save game script cleaner. Fix script instances and Delete all #. strCount = 29,471.
It seems that simply deactivating the esp does not reduce strCount at all. You must remove the related scripts but there is still a net loss in the end of 852 strings that for some reason are not removed. It seems there is yet another reason to never uninstall mods.
What Is A Safe Cushion To Start A Fresh Game?
Unfortunately, we don't know. Mod A may add 500 strings at the start of a new game and grow by 50 strings while Mod B might only start with 50 strings and grow by 500 after 100 hours. It's very tricky to gauge what would be a healthy starting point for strCount. People have reported starting a new game with a strCount of 55,000 and having corrupt saves after about 100 hours of play time.
People have voiced concern about this. StrCount probably won't grow indefinitely. At least not at a rate that would concern the majority of people. It should peak and, more or less, plateau at some point. The problem is that for some people that plateau is above 65,535 and takes 100 hours to reach
I Don't Believe You / You Are Talking Rubbish:
Try it for yourself. I have created a small mod to break your saves in a similar but much quicker fashion. In fact, I just created a new game with a less than 6.5MB save file that is corrupted. The mod simply adds a quest that runs 3 scripts that each declare 20,000 integers without even assigning them a value and then hold them in a wait loop for 30,000 seconds so that they remain persistent and are not just discarded. I was unable to create a single script with 60,000 string definitions. As the scripts either ended up on the suspended stack or did not seem to run at all. Be aware that the scripts DO NOTHING. Once the variables are declared there is very little processing going on. It's basically just 3 scripts that wait 30 seconds and decrements a counter, my commadore 64 could handle it.
60,000 variables is ridiculous of course so I created a properties version which is a closer representation of the problem. It uses 1 quest with 40 scripts, each having around 1000ish properties attached for a total of 40,000ish strings. Should be enough to break most saves unless you're not running the official DLC maybe. Note there is zero processing going on in the scripts. No loops. No events. No updates. Nothing. Just unfilled properties.
1. Download the save breaker. Variables version or Script properties version: http://www.nexusmods...im/mods/74390/?
2. Install as normal.
3. Open the game with a minimal or full load order. 60,000 (40,000 for the properties version) string definitions should break anything assuming the scripts actually run.
4. coc qasmoke from the main menu or load your favorite save.
5. Save your game.
6. Congratualtions, your save is now broken.
7. Attempt to open your save with the save game cleaner (should crash) or open the papyrus section of the save with TESV ESS Files Editor (should get "Range check error").
One interesting thing I noted from creating the variables version of the save breaker. Initially when I created the three separate scripts, all scripts had the same variable names. Eg:
Script 1 had a variable named strCountTestVariable0
Script 2 had a variable named strCountTestVariable0
Script 3 had a variable named strCountTestVariable0
The number of total strings created by this method was only 20,000 => save NOT corrupt.
I changed the scripts to each use unique variable names. Eg:
Script 1 had a variable named strCountTest1Variable0
Script 2 had a variable named strCountTest2Variable0
Script 3 had a variable named strCountTest3Variable0
This time I achieved the desired result -> Save Corrupt. And we can assume 60,000 strings were created.
The same appears to be true for script properties. There are no duplicate properties in the string table. Eg. There would otherwise be many more references to PlayerRef
It's kind of a ridiculous solution but if mod authors could use a common set of variable names / properties strCount could be reduced quite a bit.
I've added another file to the nexus download page above. This one changes the script properties version of the save breaker to use the same property names in each of the 40 scripts. It will only add 1,000 strings to your string table. It demonstrates that duplicate property names are not added to the string table. There are still 40,000 properties in the mod.
For anyone that is familiar with papyrus and phyton. How viable would it be to create a precompiler for the papyrus compiler to rename properties to use a common set of property names - Property1, Property2, etc...
Solutions - Updated 06th Aug 2016
Thanks to a talented couple of individuals there are now two solutions for this issue.
1. Crash fixes by meh321 - http://www.nexusmods...yrim/mods/72725
Crash fixes contains a workaround for saves with a string count greater than 64k. Once your save exceeds the limit, crash fixes changes the format of your save files slightly. This has two consequences for normal players:
a) you can save and load your games seamlessly and no longer care what your string count is.
b) older save game script cleaners will no longer recognise the file structure of your save and will be unable to open your saves. The most popular being:
Save game script cleaner by Hadoram & Savegame script scalpel - Disassembler - Diagnostic Tool - Papyrus Data Transfer by flexcreator.
Both of these will require an update to work with the new save format.
Thankfully, there is a new save cleaner that is able to deal with the new format:
SkyrimTools by Mark Fairchild - http://www.nexusmods...im/mods/76776/?
There are two tools contained in this mod, one of which is called Resaver. It is capable of handling files written in the new format.
Crash fixes should also make your game a lot more stable. Anyone not using Crash Fixes at this point is simply mad in my opinion.
2. SkyrimTools by Mark Fairchild - http://www.nexusmods...im/mods/76776/?
The other tool contained in this mod is called Restringer. It will allow you to keep your string count under the 65k limit. It works by taking advantage of the fact that duplicate string names do not count in the string table. Eg: there is only one "PlayerRef" in the entire string table despite it being used by scripts in many different mods.
You're going to want to know what you're doing if you use this. I'm not trying to knock it. It was the first glimmer of hope we had on the string table issue but personally, I'd have to recommend crash fixes over the restringer as it is less 'invasive' to your mod setup. The Restringer essentially rewrites scripts in what should be a predictable manor but there's always a margin for user error (restringing the wrong mod) or some unforeseen exception.
Theory On String Creation Rate:
Since properties appear to be constant as long as you are not installing more mods, uninstalling mods or updating mods then variables must be the only thing that add to string count as you play.
Open a working save in the save game cleaner & Expand script instances.
Click down through the script instances until you find one that has information in the bottom half of the window. Some script instances only have information in the top window.
Any of the "Names" you find in the bottom half will be found in the string table in TESVESSE (if you look long enough).
Notice that if there are more than one script instance with the same name it will have the same names in the bottom pane. Remember there are no duplicate entries in the string table. So...
You start a new game. Your new game has two scripts.
Script 1 has variables: Variable1, Variable2, Variable3.
Script 2 has Variables: Variable3, Variable4, Variable5.
A new instance of script 1 is created. Variable1, Variable2 and Variable3 are added to the string table.
Your string table count has increased by 3.
Another instance of script 1 is created. Nothing is added to the string table because Variable1, Variable2 and Variable3 already exist in it.
Some time later a new instance of script 2 is created. Variable4 and Variable5 are added to the string table. Variable3 already exists in the string table. - A certain amount of overlap of variable names is to be expected.
Your string table count has just increased by 2.
If the two instances of script 1 were to complete Variable1 and Variable2 would be removed from the string table. However, it is possible that more instances of script 1 would be created before the first two finish up. So string count growth seems to be attributed to script instances that are added after the game has started. The greater the number of different scripts, the higher the string count can potentially go as more scripts are instanced into the game. Because of the number of factors involved (number of scripts, time taken for script to complete, number of unique variables per script) nobody is going to be able to tell you exactly what kind of a string count cushion you are going to need. Just try to keep an eye on it as you play. If you find yourself right up near the limit quickly it is probably time to reconsider your load order.
Save File String Table vs. Game Engine String Table:
There's a question as to whether the game engine is capable of a string table count greater than 65,535. If the problem is limited to the format of the save file and the subroutines that read/write the save file, it could easier (but far from easy) to patch. If the game engine itself is not capable of a string count greater than 65,535 then patching it would be next to impossible without Bethesdas help. Conventional programming wisdom suggests that once the string pointer reaches 0xFFFF hex and is incremented it would reset to 0x0000 and begin overwriting the existing contents of the string table. We know that this is definitely the case for the save file (corrupted saves) but is it the same for the game engine? To try to determine this I created another version of the save breaking mod. It uses 11 scripts, each with 9,000 string variables for a total of 99,000 strings. The scripts output the contents of their first and last variable (String0 and string8999) to the papyrus log every 5 seconds. If you install it on to an existing save with a string count of 32,070 or higher it should be enough to fill the table to the end, wrap around and overwrite the entire string table again:
Minimum existing string table size for test: 32,070.
Strings needed to fill the table to 0xFFFF: 65,535 - 32,070 = 33,465.
Strings remaining for pointer wrap around: 99,000 - 33,465 = 65,535 - ie. enough to overwrite the entire table again.
We know that the save file can not handle this many strings but if the game engine is not capable of a string count higher than 65,535 as well, you would expect there to be serious problems if the existing string table was overwritten completely. CTDs, massive papyrus log spam or mods just not working in general. If the game pointer was 16 bits there are only a couple of things that could happen that I can think of:
1. Wrap around to 0x0000 as discussed above which should result in broken mods and MCMs, CTDs, and/or papyrus log spam.
2. 'Stick' at 0xFFFF in which case strings added above 65,535 would be affected which would mean the 99,000 string variables added by the mod would be broken.
I don't see either of these results with the mod installed. No CTDs, MCMs and mods appear to be working fine and no massive papyrus spam that you would expect. The contents of the first and last script variables also report back correctly in the logs. It would SEEM to suggest that the game engine is capable of a string count higher than 65,535. Don't just take my word on it. Try it for yourself and report back.
One caveat on the mod. If you are not seeing debug.notification spam in the top left corner of your screen after loading the mod the scripts are probably not running. They sometimes end up on the suspended stack as they might be too much for papyrus to swallow. If this is the case any save you make at this point should work ok because the variables have not been added to the string table. If you load the save in the save game cleaner you will find the scripts on the suspended stack.
I did not start this post to bash any mods. I was simply using them to demonstrate a point and the shortcomings stated here are NOTHING to do with the mods mentioned. Interesting NPCs and Illustrious Cloaks Of Skyrim are both excellent mods and you should download them now:
Interesting NPCs by Kris Takahashi: http://www.nexusmods...rim/mods/8429/?
Illustrious HDT Cloaks of Skyrim by Euphy aka EuphoricKnave: http://www.nexusmods...im/mods/72581/?
If you have any information you feel I have left out regarding this issue let me know and I will update the post.
Edited by bcsp, 06 August 2016 - 01:15 PM.