Corrupt Saves - StrCount > 0xFFFF - CTD on load

[bcsp]

I'm posting here as this issue could use some more exposure and I fear the timing of it's discovery with the release of Fallout 4 will mean it will go unaddressed if left much longer. I see it as a pretty serious threat to the longevity of Skyrim in general. It's a bit of a long, technical read but please try to bear with me and take the time to digest the post fully as there is a lot of information. I feel there are a lot more people out there have this issue and don't realize it because your first instinct would be that it's a mod conflict of some sort.

 

Issue Summary:

A limitation in Skyrims save file format has been discovered which causes saves to become unloadable - CTD on loading these affected saves. This is NOT random CTD on load that Skyrim does every now and then. These saves will never load.

 

This issue has been reported on:

LL: http://www.loverslab.com/topic/56588-corrupted-saves/page-1

STEP forum: http://forum.step-project.com/topic/9683-i-found-a-new-bug-or-maybe-is-a-not-solvable-old-one-in-the-save-files/

Bethesda forum: http://forums.bethsoft.com/topic/1568922-wipz-skyrim-script-extender-skse/

and the SKSE team have been emailed about it.

 

Am I Affected?

To test whether or not you have this issue.

1. Download the Save game script cleaner by Hadoram: http://www.nexusmods.com/skyrim/mods/52363/?

2. Open the save giving you trouble.

3. If the script cleaner immediately crashes it's likely this particular save has this issue.

4. If it does not crash. Happy days. You are not yet affected by this issue. Please try searching other threads regarding CTDs on load as this particular flavour of 'CTD on load' is very specific

 

How Close Are My Saves To Imploding?

1. Download TESV ESS Files Editor by fubrus: http://www.nexusmods.com/skyrim/mods/27119/?

2. Open your suspect save.

3. Expand "Global data table 3"

4. Click on Papyrus (don't need to expand).

5. If you receive a pop up error "Range check error" your save is almost definitely screwed.

6. Note the "Strings" entry.

7. 65,535 - (the number of strings from step 6) = the cushion you have from having corrupt saves.

 

Technicial Issue Description:

Please see the following link: http://www.uesp.net/wiki/Tes5Mod:Save_File_Format/Papyrus

 

Note the second entry

strCount uint16

strCount is a 16 bit field. It's maximum value is FFFF hex or 65,535 decimal. Once the number of strings is incremented past 0xFFFF we're guessing the pointer 'wraps around' and resets to 0x0000 making the save unreadable. I've seen it happen where save 1 is corrupt and save 2 (5 mins later) is fine. It would seem to support the theory that the pointer is being incremented past 0xFFFF and then decremented again below 0xFFFF as the game is played and strings are created and discarded at the very limit of strCount. Corruption this way can occur in game while you are oblivious you are creating bad saves. IE. the game does not CTD nor is there any indication of any weirdness within papyrus while playing so it seems that the game engine itself is capable of strCount > 0xFFFF but the save file itself has become unfit for purpose.

 

Just to put it in context: Vanilla Skyrim + Dawnguard + Hearthfires + Dragonborn + Unofficial Legendary Patch + SkyUI + Live Another Life = A total of 28,619 strings. Nearly half of the field (43%) is consumed with an almost vanilla setup and that's a fresh game, sitting in the LAL cell. As far as we can figure, as more and more scripts are instanced as the game is played, more strings are added and strCount grows. I would love to have a figure for strCount for this vanilla setup after say 100 hours of play but can not afford the time right now.

 

Something to bear in mind is that this is not a performance issue. It is simply a capacity issue. Some of the heaviest mods, string-wise (through no fault of their own), are not performance intensive at all. For example and I'm only using this mod as an example as people will be familiar with it and probably surprised with it's impact. Interesting NPCs consumes 3,471 strings (over 5% for a single mod) in a new game, again, just sitting in the LAL cell.

 

What Are Strings?

They appear to be:

• variable definitions for functions using local variables. Usually they are removed from the string table once a function has completed but some of them are persistent - MCM variables.
• script properties. The vast majority of the string table look like properties.
• script/fragment names minus the .pex file extension. These script names actually appear to be properties as well. It explains why there are very few script names appearing in the string table.

 

What Type Of Mods Have The Biggest Impact On strCount?

• Mods with a large number of scripts . This is not necessarily true. But as a general rule of thumb, more scripts = more properties. It seems this is why large dialog mods (Interesting NPCs) have such a large impact. I'm no expert on modding dialogue but apparently scripts are required to simply advance quests stages. To me, it seems like a very, very DUMB method implemented by Bethesda for mod authors to add DIALOG to a RPG and then slap a 16 bit limit on it.
• Mods with expansive complex MCMs. Each MCM entry (slider, toggle, whatever) requires several variable definitions. Scrolling through the strCount table I see lots of variables with 'OID' in their name and they're just the variables from mod authors that use that convention. I sigh when I see mods adding pointless MCMs. Keep it simple folks.
• Scripts with lots of properties. Properties seem to be the biggest contributor to the size of the string table. For Eg. I recently 'down-graded' the excellent "Illustrious Cloaks Of Skyrim" from the scripted version to the crafting version. Scripted version has around 230 properties in one script. While the scripted version is a much more elegant and proper implementation, the crafting version has no scripts at all.
• Animation mods. Especially problematic for users on 'that other site' as each animation has been reported to add several strings.

Impact Of Mod Uninstallation:

Vanilla setup from above. strCount = 28,619.

Install interesting NPCs. strCount = 32,090.

Deactivate Interesting NPCs esp. strCount = 32,090.

Deactivate Interesting NPCs mod entirely (ie. remove scripts). strCount = 29,471.

Run the save through the save game script cleaner. Fix script instances and Delete all #. strCount = 29,471.

 

It seems that simply deactivating the esp does not reduce strCount at all. You must remove the related scripts but there is still a net loss in the end of 852 strings that for some reason are not removed. It seems there is yet another reason to never uninstall mods.

 

What Is A Safe Cushion To Start A Fresh Game?

Unfortunately, we don't know. Mod A may add 500 strings at the start of a new game and grow by 50 strings while Mod B might only start with 50 strings and grow by 500 after 100 hours. It's very tricky to gauge what would be a healthy starting point for strCount. People have reported starting a new game with a strCount of 55,000 and having corrupt saves after about 100 hours of play time.

 

People have voiced concern about this. StrCount probably won't grow indefinitely. At least not at a rate that would concern the majority of people. It should peak and, more or less, plateau at some point. The problem is that for some people that plateau is above 65,535 and takes 100 hours to reach

 

I Don't Believe You / You Are Talking Rubbish:

Try it for yourself. I have created a small mod to break your saves in a similar but much quicker fashion. In fact, I just created a new game with a less than 6.5MB save file that is corrupted. The mod simply adds a quest that runs 3 scripts that each declare 20,000 integers without even assigning them a value and then hold them in a wait loop for 30,000 seconds so that they remain persistent and are not just discarded. I was unable to create a single script with 60,000 string definitions. As the scripts either ended up on the suspended stack or did not seem to run at all. Be aware that the scripts DO NOTHING. Once the variables are declared there is very little processing going on. It's basically just 3 scripts that wait 30 seconds and decrements a counter, my commadore 64 could handle it.

 

60,000 variables is ridiculous of course so I created a properties version which is a closer representation of the problem. It uses 1 quest with 40 scripts, each having around 1000ish properties attached for a total of 40,000ish strings. Should be enough to break most saves unless you're not running the official DLC maybe. Note there is zero processing going on in the scripts. No loops. No events. No updates. Nothing. Just unfilled properties.

 

1. Download the save breaker. Variables version or Script properties version: http://www.nexusmods.com/skyrim/mods/74390/?

2. Install as normal.

3. Open the game with a minimal or full load order. 60,000 (40,000 for the properties version) string definitions should break anything assuming the scripts actually run.

4. coc qasmoke from the main menu or load your favorite save.

5. Save your game.

6. Congratualtions, your save is now broken.

7. Attempt to open your save with the save game cleaner (should crash) or open the papyrus section of the save with TESV ESS Files Editor (should get "Range check error").

 

One interesting thing I noted from creating the variables version of the save breaker. Initially when I created the three separate scripts, all scripts had the same variable names. Eg:

 

Script 1 had a variable named strCountTestVariable0

Script 2 had a variable named strCountTestVariable0

Script 3 had a variable named strCountTestVariable0

The number of total strings created by this method was only 20,000 => save NOT corrupt.

I changed the scripts to each use unique variable names. Eg:

 

Script 1 had a variable named strCountTest1Variable0

Script 2 had a variable named strCountTest2Variable0

Script 3 had a variable named strCountTest3Variable0

This time I achieved the desired result -> Save Corrupt. And we can assume 60,000 strings were created.

The same appears to be true for script properties. There are no duplicate properties in the string table. Eg. There would otherwise be many more references to PlayerRef

It's kind of a ridiculous solution but if mod authors could use a common set of variable names / properties strCount could be reduced quite a bit.

 

I've added another file to the nexus download page above. This one changes the script properties version of the save breaker to use the same property names in each of the 40 scripts. It will only add 1,000 strings to your string table. It demonstrates that duplicate property names are not added to the string table. There are still 40,000 properties in the mod.

 

For anyone that is familiar with papyrus and phyton. How viable would it be to create a precompiler for the papyrus compiler to rename properties to use a common set of property names - Property1, Property2, etc...

 

Solutions - Updated 06th Aug 2016

 

Thanks to a talented couple of individuals there are now two solutions for this issue.

 

1. Crash fixes by meh321 - http://www.nexusmods.com/skyrim/mods/72725

Crash fixes contains a workaround for saves with a string count greater than 64k. Once your save exceeds the limit, crash fixes changes the format of your save files slightly. This has two consequences for normal players:

a) you can save and load your games seamlessly and no longer care what your string count is.

b) older save game script cleaners will no longer recognise the file structure of your save and will be unable to open your saves. The most popular being:

Save game script cleaner by Hadoram & Savegame script scalpel - Disassembler - Diagnostic Tool - Papyrus Data Transfer by flexcreator.

Both of these will require an update to work with the new save format.

 

Thankfully, there is a new save cleaner that is able to deal with the new format:

SkyrimTools by Mark Fairchild - http://www.nexusmods.com/skyrim/mods/76776/?

There are two tools contained in this mod, one of which is called Resaver. It is capable of handling files written in the new format.

 

Crash fixes should also make your game a lot more stable. Anyone not using Crash Fixes at this point is simply mad in my opinion.

 

2. SkyrimTools by Mark Fairchild - http://www.nexusmods.com/skyrim/mods/76776/?

The other tool contained in this mod is called Restringer. It will allow you to keep your string count under the 65k limit. It works by taking advantage of the fact that duplicate string names do not count in the string table. Eg: there is only one "PlayerRef" in the entire string table despite it being used by scripts in many different mods.

 

You're going to want to know what you're doing if you use this. I'm not trying to knock it. It was the first glimmer of hope we had on the string table issue but personally, I'd have to recommend crash fixes over the restringer as it is less 'invasive' to your mod setup. The Restringer essentially rewrites scripts in what should be a predictable manor but there's always a margin for user error (restringing the wrong mod) or some unforeseen exception.

 

 

 

Unfortunately, There is no quick and easy solution yet. Short of reducing your load order and testing for 100 hours each time...

 

The ideal solution is for strCount to be changed to a 32 bit field. I think Bethesda could fix this in 15 mins with access to the source code and simply redefine the strCount field as a double word but Beth is too busy making money. They are a business I understand but I find It ironic that the issue is more likely to be fixed by hacker/cracker type of people than Bethesda. At this point we need people versed in assembler code and ideally, familiar with the tesv.exe binary.

 

Some notable patches have been created that may drastically reduce your string count if you use these mods:

DynDoLod: String count has been reduced dramatically from versions 1.48 and up.

SL 1.61 (If you don't know what that is you don't need it): Huge drop in string counts reported by using markdfs jcontainers patch. The patch requires jcontainers 3.3: http://www.loverslab.com/topic/56588-corrupted-saves/?p=1520136

 

To help you make a more considered approach to your load order EpitomyofShyness has compiled an online spreadsheet with the reported string counts of various mods: https://docs.google....dit?usp=sharing

Please be careful when adding information to the document.

 

For modders the current methods to help reduce string count are:

Use duplicate property/variable names as much as possible.

Export variables to external storage.

 

 

Theory On String Creation Rate:

Since properties appear to be constant as long as you are not installing more mods, uninstalling mods or updating mods then variables must be the only thing that add to string count as you play.

 

Open a working save in the save game cleaner & Expand script instances.

Click down through the script instances until you find one that has information in the bottom half of the window. Some script instances only have information in the top window.

Any of the "Names" you find in the bottom half will be found in the string table in TESVESSE (if you look long enough).

Notice that if there are more than one script instance with the same name it will have the same names in the bottom pane. Remember there are no duplicate entries in the string table. So...

 

Simple example:

You start a new game. Your new game has two scripts.

Script 1 has variables: Variable1, Variable2, Variable3.

Script 2 has Variables: Variable3, Variable4, Variable5.

 

A new instance of script 1 is created. Variable1, Variable2 and Variable3 are added to the string table.

Your string table count has increased by 3.

Another instance of script 1 is created. Nothing is added to the string table because Variable1, Variable2 and Variable3 already exist in it.

Some time later a new instance of script 2 is created. Variable4 and Variable5 are added to the string table. Variable3 already exists in the string table. - A certain amount of overlap of variable names is to be expected.

 

Your string table count has just increased by 2.

 

If the two instances of script 1 were to complete Variable1 and Variable2 would be removed from the string table. However, it is possible that more instances of script 1 would be created before the first two finish up. So string count growth seems to be attributed to script instances that are added after the game has started. The greater the number of different scripts, the higher the string count can potentially go as more scripts are instanced into the game. Because of the number of factors involved (number of scripts, time taken for script to complete, number of unique variables per script) nobody is going to be able to tell you exactly what kind of a string count cushion you are going to need. Just try to keep an eye on it as you play. If you find yourself right up near the limit quickly it is probably time to reconsider your load order.

 

Save File String Table vs. Game Engine String Table:

There's a question as to whether the game engine is capable of a string table count greater than 65,535. If the problem is limited to the format of the save file and the subroutines that read/write the save file, it could easier (but far from easy) to patch. If the game engine itself is not capable of a string count greater than 65,535 then patching it would be next to impossible without Bethesdas help. Conventional programming wisdom suggests that once the string pointer reaches 0xFFFF hex and is incremented it would reset to 0x0000 and begin overwriting the existing contents of the string table. We know that this is definitely the case for the save file (corrupted saves) but is it the same for the game engine? To try to determine this I created another version of the save breaking mod. It uses 11 scripts, each with 9,000 string variables for a total of 99,000 strings. The scripts output the contents of their first and last variable (String0 and string8999) to the papyrus log every 5 seconds. If you install it on to an existing save with a string count of 32,070 or higher it should be enough to fill the table to the end, wrap around and overwrite the entire string table again:

 

Minimum existing string table size for test: 32,070.

Strings needed to fill the table to 0xFFFF: 65,535 - 32,070 = 33,465.

Strings remaining for pointer wrap around: 99,000 - 33,465 = 65,535 - ie. enough to overwrite the entire table again.

 

We know that the save file can not handle this many strings but if the game engine is not capable of a string count higher than 65,535 as well, you would expect there to be serious problems if the existing string table was overwritten completely. CTDs, massive papyrus log spam or mods just not working in general. If the game pointer was 16 bits there are only a couple of things that could happen that I can think of:

 

1. Wrap around to 0x0000 as discussed above which should result in broken mods and MCMs, CTDs, and/or papyrus log spam.

2. 'Stick' at 0xFFFF in which case strings added above 65,535 would be affected which would mean the 99,000 string variables added by the mod would be broken.

 

I don't see either of these results with the mod installed. No CTDs, MCMs and mods appear to be working fine and no massive papyrus spam that you would expect. The contents of the first and last script variables also report back correctly in the logs. It would SEEM to suggest that the game engine is capable of a string count higher than 65,535. Don't just take my word on it. Try it for yourself and report back.

 

One caveat on the mod. If you are not seeing debug.notification spam in the top left corner of your screen after loading the mod the scripts are probably not running. They sometimes end up on the suspended stack as they might be too much for papyrus to swallow. If this is the case any save you make at this point should work ok because the variables have not been added to the string table. If you load the save in the save game cleaner you will find the scripts on the suspended stack.

 

End Note:

I did not start this post to bash any mods. I was simply using them to demonstrate a point and the shortcomings stated here are NOTHING to do with the mods mentioned. Interesting NPCs and Illustrious Cloaks Of Skyrim are both excellent mods and you should download them now:

Interesting NPCs by Kris Takahashi: http://www.nexusmods.com/skyrim/mods/8429/?

Illustrious HDT Cloaks of Skyrim by Euphy aka EuphoricKnave: http://www.nexusmods.com/skyrim/mods/72581/?

 

If you have any information you feel I have left out regarding this issue let me know and I will update the post.

Edited August 6, 2016 by bcsp

[jones177]

Hi

 

The problem is there are people like me that have lots of the script heavy mods installed, have had Skyrim maxed out with mods for years & that have never had an issues of saves not loading unless there is a missing master.

 

I have kept 3 saves of each character that I have created since the game came out (11 gbs in the same folder). All will load. Even the 28mb, 70+ level character saves (700+ hour). Some don't last long because they are dependent on mods that are no longer on my load list(skyre - perkus maximus) but they will load.

 

If the save proses is interrupted by a CTD the save will be corrupt because it is incomplete. It will not load & not clean. The game will tell you that it is corrupt in the load window. That has only happened twice in the silly amount of hours I have played.

 

As for "Save game script cleaner"' my older saves crash on the current version but not on the older versions. The author even notes this on the download page. I keep several versions for this reason.

 

I use over 250 mods & probably about 50 use scripts & script lag is the only issue that I have had with these types of mods. It don't allow me to play close to 30fps & that was my frame rate at 4k with ENB (up until recently).

 

I think my game would become unplayable from the lag long before the saves corrupt.

 

We need to hear from people that run 300+ mods with 70+ script mods & a 1000+ hours on a save to see if they have reached a limit. I know I haven't

 

 

Later

[malilie]

 

We need to hear from people that run 300+ mods with 70+ script mods & a 1000+ hours on a save to see if they have reached a limit. I know I haven't

 

 

Later

 

I have. And it's quite easily reproducable. You could be using 1,000 script heavy mods and it wouldn't necessarily cause your save to have this problem. It's not the number of scripts, it's the number of strings.

[bcsp]

Please don't be so dismissive. I didn't start modding yesterday.

 

Missing masters are easy to detect these days. SKSE will even tell you what you are missing. Or load your full load order in TesEdit and it will tell you.

Obviously the game is not CTDing during the save process, you're not giving me much credit here.

Like I've said in the OP. This is not a performance issue. I'd wager 99.9% of these strings are not in active use at any one time. There's a difference between processing variables and storing variables. And save corruption can happen in the LAL cell at 60fps with script latency well below 50ms.

 

Can you open one of these 1000 hour saves in TESV ESS Files Editor (not the save cleaner) and tell me what the strcount is please?

[bcsp]

Updated main post with "I Don't Believe You / You Are Talking Rubbish" section for people of little faith.

Now you can break your own saves. Don't thank me all at once ;)

Edited March 21, 2016 by bcsp

[archtrtd1]

Thank you so much for sharing this information with us. I believe you. I have had a rather lofty dream of someday, maybe finding the right combination of mods that I enjoy playing and are stable enough for an entire play-through. I've only managed to get to about level 27 before I really started to have problems (instant ctds, games not loading, horrible stuttering, etc). I stumbled upon the TESV ESS Editor a couple of weeks ago when I was searching for information on the effects of scripts on game saves. My level 27 character (with a ton of scripted/MCM mods) had 41341 strings, 6107 scripts, and 122002 script instances.

 

I have had a difficult time finding solid/useful information on this subject. I would really appreciate if someone could explain the following (please forgive my ignorance):

1. What is the relationship between strings and scripts?

2. Is there any clear documentation on how to interpret the information generated by TESV ESS Editor?

 

 

Also, have you posted this information on the Bethesda forums?

 

Thanks.

[bcsp]

The number of strings isn't detrimental at all until you pass the magic number 65,535.

I'm sorry but it looks like your problems aren't related to the strCount > 0xFFFF issue as your strCount is well below the limit.

strCount overflow also does not symptomize as stutter or CTDs while you're playing the game. Only CTDs while loading a corrupted save that can be identified in the "Am I Affected" and the "How Close Is My Save To Imploding" sections.

[archtrtd1]

I understand that my problems currently are not related to the strCount > 0xFFFF issue. I'm thinking long term. How long would it take, with a particular combination of mods, to reach the threshold where my game saves are ruined? That is what I am trying to troubleshoot.

[bcsp]

I don't think strCount will continue to grow. Skyrim seems to be intelligent enough to release strings that aren't being used. Problem is many of them are persistent (script properties for example).

I believe strCount will more or less plateau at some point. The problem is that for some people that platau is above 65,535 and takes 100 hours to reach. Finding out your save is screwed after 100 hours of play sucks a serious amount of ass.

[archtrtd1]

I totally agree! It would really suck. I am doing my best to figure out how to prevent it from happening to me. :) It's good to know that there is a point where the number will plateau! I was worried that it would keep adding up over time.