Corrupt Saves - StrCount > 0xFFFF - CTD on load

[Mookeylama]

i'm trying to test some of the merges i've put together to see if they might be big offenders. what's the best way to test? can u just uncheck a plugin, load the save, save again, then check string count? or do u need to run Save Cleaner

[Rubbermatt]

 

"

bcsp, just a thought, and this may seem a bit extreme, but would it be possible to modify your save breaker to spawn over 4.3 G strings? This might make it possible to confirm whether the game engine is capable of handling up to 32 bits of strings.

yeah we need to find out if it's a game engine limitation. if not, then there might be hope for an SKSE fix

 

oh and u guys saw the Sexlab patch made for this problem recently? it lowered the string count by 50%. from 2000 to about 1000. we need something like it for Interesting NPCS and Legacy of Dragonborn

 

 

Interesting NPCs has something like 6000 scripts, that's not getting changed in a hurry.

Let's be realistic here, the best we can do is let modders and mod creators know that this limitation exists in the game, it's hardcoded into the game and at this point the best known work around is using json.

Provide as much information on string count in as many mods as possible so that modders can make an informed choice of which mods to use for their playthrough.

(e.g. someone may decide that a high impact mod is something that they can't live without, but knowing that there is a hard limit to strcount they then round out their mod list with low impact mods.)

[Mookeylama]

i haven't updated to USLEEP yet as some large merges i made rely on the individual uskp. i was gonna update next play.

anyone know if there's a significant change in strings (+ or -) between the separate uskp plugins and USLEEP?

[lynnyfm]

I'm not versed in modding or even programming, so all I can contribute is that my saves started corrupting a few days ago. I drove myself crazy trying to figure out how I had broken my game. Thanks to everyone who has contributed thus far. With the DyndoLOD update and some selective pruning/replacing after reading the Google doc, the count is down about 10000, my game is saving just fine again, and at least now I can keep an eye on it.

[Gawad]

Hi all, I'm trying to get a better grasp on how to get a more accurate string count than just loading up a mod into a fresh save, saving again and using TESV ESS to look at the before-after string count. This is because not all strings appear to get included when a mod is first initialized, and a good number of them seem to show up as a mod is "put through its paces" so to speak. But this seems to depend on the mod and is very unpredictable just by working with a save game.

So....

I thought I'd have a look at the properties in the scripts of a mod itself. Thought I'd use Simply Knock as an example, as 1) I trust the author codes properly, 2) it's not too big, 3) is known to increase strcount by roughly 100 upon initialization.

Basically, I'm thinking of trying to total up all the unique strings that the mod could contribute in order to produce a "worst case" for that mod. That way you wouldn't have to worry about whether that mod would eventually put you over the 16-bit strcount limit, especially if you're considering adding such a mod to a game that seems to have relatively plateaued its strcount.

Now, using Chesko's _sk_dooraliasscript as an example, using Notepad++ the source .psc doesn't show any strings, and the .pex, being compiled, is gobbledygook, obviously. However, if I decompile the _sk_dooraliasscript.pex file using PEXD with the Write string table option selected, the resulting _sk_dooraliasscript.dpex file has a .STRINGS list that looks like this in Notepad++ (1 is just a whitespace):

0 _SK_DoorAliasScript

1

2 GetState

3 GotoState

4 OnLockStateChanged

5 DebugLog

6 hidden

7 conditional

8 ReferenceAlias

9 ::PlayerRef_var

10 actor

11 ::_SK_Setting_LogLevel_var

12 globalvariable

13 PlayerRef

14 _SK_Setting_LogLevel

15 None

16 Function that switches this object to the specified state

17 newState

18 String

19 ::NoneVar

20 onEndState

21 self

22 ::State

23 onBeginState

24 Function that returns the current state

25 aiSeverity

26 Int

27 asLogMessage

28 ::temp5

29 ::temp6

30 Bool

31 LOG_LEVEL

32 ::temp7

33 ::temp9

34 ::temp8

35 GetValueInt

36 [simplyKnock][Debug]

37 debug

38 trace

39 [simplyKnock][info]

40 [simplyKnock][Warning]

41 [simplyKnock][ERROR]

42 ::temp0

43 objectreference

44 ::temp1

45 the_door

46 ::temp2

47 cell

48 ::temp3

49 ::temp4

50 GetRef

51 IsLocked

52 GetParentCell

53 Linked door was re-locked with player in cell. Unlocking door as owner.

54 Lock

Now, here's my question: if I were to decompile all 25 files in Simply Knock into .dpex files and total up entries that are unique across the mod's .STRINGS lists, can it be assumed that this would represent the maximum number of strings Simply Knock could potentially ever contribute to a save file? Or am I missing something stupidly obvious???

Also, if this is correct, does anybody have any idea how I might go about automating the process??? I mean, apart from dumping each list into Excel and then filtering for unique values...which isn't automation so much.

Thanks for any info at all!

Oh, and one more thing, I was wondering just how "opaque" is a mod's total srtcount contribution is to its author?

Edited April 13, 2016 by Gawad

[Gawad]

O.k., so brief update (there went my lunch hour:)

 

Using my former example of Simply Knock.... Combining all the strings in all the decompiled pex files in the mod shows 1845 total strings when dumped into a single Excel column, not including the script name and following whitespace. Removeing duplicates leaves 982 unique entries, including variables such as "::TimeToGo_var", and other values such as ": Goodbye!', ". We're friends!", "::Config_var" and "::temp0", etc.

 

Does this sound plausible and/or about right to anyone in the know?

 

Also, I assume, you would never get anywhere near 982 added strcount (even if this is correct) as garbage collection would remove some as Simply Knock's scripts run their course.

Edited April 13, 2016 by Gawad

[DavidJCobb]

Looks plausible. I've been building my own save tool in my spare time (usually to aid debugging for other projects) and have seen similarly-formatted names embedded in savegames. I do know that function docstrings are stored as well, though probably only while the functions are executing; removing them could yield a small benefit. (They're not actually used for anything that helps us modders, so replacing them with comments oughta be fine.)

 

The numbered temp vars are used for compound operations, such as DoSomething(!DoSomethingElse()). These operations need to be done one by one, so these temp vars are used to hold results. You'll see them recur across a lot of mods.

Edited April 13, 2016 by DavidJCobb

[EpitomyofShyness]

If you test more mods like this I can add in a "maximum potential string count" column in my spreadsheet, with the explanation that a player is highly unlikely to ever actually hit that maximum but its just one more thing to be aware of.

[Gawad]

If you test more mods like this I can add in a "maximum potential string count" column in my spreadsheet, with the explanation that a player is highly unlikely to ever actually hit that maximum but its just one more thing to be aware of.

I'm not yet convinced that's the best thing to do, even with a warning. In fact right now, I think it might even be counter productive. Might need to test at least a few more mods this way....I wasn't expecting the results I got with Simply Knock, and I'm far, far from sure how to interpret them.

 

I didn't think the total unique number of strings for Simply Knock would be that high. Another reason I ran a test with Simply Knock is that it's already in your spreadsheet, with string count values taken just after initialization–I assume, since there's no strict testing methodology. Your spreadsheet lists counts from ~75 to ~175, so lets call it 125 for the heck of it. Now, I find 125 a little on the high side for a mod like Simply Knock, but if I were playing a thief I might forgive that given it would be so handy, install it and be on my merry way, only breaking into unoccupied homes. Great.

 

Thing is, if word starts going 'round that the mod can ramp up your string count by nearly 1000 (!) I'm concerned that a lot of people might forego using it for no justifiable reason. I have no real basis from which to say this, but I just have a really hard time believing that Simply Knock has any chance of pushing your strcount anywhere near 1000 (I mean, that's huge for a mod this size and that only comes into play when you interact with a door), and suggesting such a thing is very premature, I think. (I might even say "alarmist")

 

I just wish there was a way to check that the mod is actually gradually driving your string count from 125 to 1000 over time before declaring something like that, otherwise it could be qualified as just fear-mongering.

 

So, basically, yeah, I wanted an example of how a "worst case scenario" would look, and now I'm like, "wait, this can't be right..."

 

Then again, Immersive Armors adds 1000 just at initialization and I wouldn't have believed it unless I'd tried it myself. And, yup, I uninstalled it, but on the basis of those 1000 strings I immediately lost.

 

Damn, what a confounding PITA!

Edited April 13, 2016 by Gawad

[Gawad]

One more test: Trade & Barter. This is a really small mod, very popular, considered by many to be the "Holy Grail" of economy mods. Only 3 scripts in this mod, but string count averages 350 at initialization. Totalling the strings from the 3 scripts: 775. Removing duplicates leaves 657, and when they're sorted you can see that a lot of these relate to its MCM. The MCM script alone has 636 strings. So, yeah, in this case I'd expect T&B to go in heavy (it does), but not grow string count much over time. From what I've understood, since strings aren't duplicated in the save's string table, common ones like "GetState" and "GoToState" won't grow your count since they're shared across a number of mods, so T&B might grow your count by, what? maybe another 200 for a total of 550? Less if you don't fiddle with its MCM over time? Just a guesstimate, but at least this one is making more sense to me than Simply Knock.

Edited April 13, 2016 by Gawad