Corrupt Saves - StrCount > 0xFFFF - CTD on load

[EpitomyofShyness]

 

If you test more mods like this I can add in a "maximum potential string count" column in my spreadsheet, with the explanation that a player is highly unlikely to ever actually hit that maximum but its just one more thing to be aware of.

 

I'm not yet convinced that's the best thing to do, even with a warning. In fact right now, I think it might even be counter productive.... Thing is, if word starts going 'round that the mod can ramp up your string count by nearly 1000 (!) I'm concerned that a lot of people might forego using it for no justifiable reason. I have no real basis from which to say this, but I just have a really hard time believing that Simply Knock has any chance of pushing your strcount anywhere near 1000 (I mean, that's huge for a mod this size and that only comes into play when you interact with a door), and suggesting such a thing is very premature, I think. (I might even say "alarmist")

 

 

Yeah your probably right. I'll keep track of the data, and once we have a better grasp on it all it might be time to start tracking "potential for increase" but certainly not right now.

[cdcooley]

If you're going to do an estimated maximum you really need to get the base string cache of a new game so you can filter out any strings that are already used by the game itself (basically all of the names for the game's core scripts, properties, temporary variables, and string constants). And that dump of strings from the PEX file will include strings from variables and strings inside functions which aren't typically stored in the save game (unless that particular function was running at the time of the save) so it's certainly a misleading upper limit. But on the other side you'll also never be able to get more than an estimate because mods that set their string or string array values at run-time instead of being filled in the CK won't show up in those string cache lists. The worse case of that is that the same small script with just one string variable could be attached to thousands of game objects so you can't even just count the string variables. Fortunately most mods won't be doing that sort of thing so it's not something you really need to worry about when creating an estimate.

 

 

A reasonable estimate for actual string use by most mods is probably just the count of unique global variable and property names in everything but the perk, scene, and topic info fragment scripts plus the number of global string and string array variables/properties. Unfortunately since the string array properties could be filled in the CK or via code in the script there's no good way to estimate how many values are in each of those arrays. At any given moment the mod probably is using a few more strings because active functions temporarily add to the string cache, but because they are temporary you wouldn't really see them consistently being stored in the save game file.

 

If someone has the time it would be interesting to get a count of those global name counts for a few of the mods that people have already reported.

 

I can account for 23 new strings added by the InigoMCM scripts (not used by the game or Inigo himself) and I know one more string is added by the InigoMCM after hovering over an item and getting the info box help text at the bottom of the MCM. So I can literally count the 23-24 strings added by that mod.

 

I also did a check through the source for Inigo's scripts. After filtering out the obvious game duplicates (like PlayerRef) I get a count of 98 unique names and 10 string variable values for a total of 118. I'm sure there are at least a few more strings in that list that duplicate names used by the game so the value someone reported of 113 strings seems perfectly reasonable. And Inigo does have scripts that run periodically so at any given point it's also reasonable a save might have 10-20 extra values depending on which of the functions was running (or maybe more if more than one function happens to trigger at the same time).

 

I've also checked my StorageHelpers mod. Initially it adds 68 strings, then 20 more can be added if you craft the various kinds of containers and then every named container you set up adds one more string (with the name you give the container).

 

I would be very curious to know if the same holds true for other mods.

[TerraKitsune]

G strings hehe.

I see what you mean though. See will the game break over the 32 bit limit. 99,000 took some time to make, 4.3 million.... I think I'd need a month off work :wink:

Not million. Billion. 2^16-1 = 65535, 2^32-1 = 4,294,967,295 Skyrim would implode long before that from running out of memory.

[bcsp]

Whoops forgot to carry the one ;)

Yea that version of the save breaker won't get made.

[TerraKitsune]

What about arrays? If you have say..

String mystr = "Hello world!"

String myweather = "It's raining again!"


int i = 0
int
 j = 10
int
 s = 2
while (i < j)

  myPrint( mystr )

  myPrint( myweather )

  i = i + s

endWhile

and instead write it like (syntax may not be fully accurate but you get the point)

String[ ] myStrings = new String[50]

Int[ ] myInts = new Int[50]



myStrings[1] = "Hello world!"

myStrings[2] = "It's raining again!"


myInts [1] = 0

myInts [2]  = 10

myInts[3] = 2


while (myInts[1] < myInts[2])

  myPrint( myStrings[1] )

  myPrint( myStrings[2] )

  myInts[1] = myInts[1] + myInts[3]

endWhile

How many strings would you get? Properties can be put into array as well, although that would make you go through extra mile on CK.

 

Obviously readability would suffer, but there's ways around that. For scripts it should be even possible to make a precompiler that goes through the source and swaps standard variables into arrays - IF that helps with string count. And what about performance side - are arrays significantly slower than normal variables?

Edited April 18, 2016 by TerraKitsune

[Gawad]

What about arrays? If you have say..

 

...

 

How many strings would you get? Properties can be put into array as well, although that would make you go through extra mile on CK.

 

Obviously readability would suffer, but there's ways around that. For scripts it should be even possible to make a precompiler that goes through the source and swaps standard variables into arrays - IF that helps with string count. And what about performance side - are arrays significantly slower than normal variables?

 

markdf over on LL is developing a tool that does pretty much exactly that: moves a papyrus script's properties to jarrays. He's looking at having it work like a Skyproccer. He did point out, however, that mods patched this way can't act as masters to any other mods, as those mods wouldn't be able to find their master's properties. So that would remain one important limitation.

 

Oh and about that 32-bit save breaker...euh, yeah, 32-bits worth of strings on a 32-bit limited program...that really was a brilliant suggestion, not. :huh:

Edited April 18, 2016 by Gawad

[TerraKitsune]

 

 

markdf over on LL is developing a tool that does pretty much exactly that: moves a papyrus script's properties to jarrays. He's looking at having it work like a Skyproccer. He did point out, however, that mods patched this way can't act as masters to any other mods, as those mods wouldn't be able to find their master's properties. So that would remain one important limitation.

 

Oh and about that 32-bit save breaker...euh, yeah, 32-bits worth of strings on a 32-bit limited program...that really was a brilliant suggestion, not. :huh:

 

 

Hmm... is that really a problem? In regards to quality of code and such, normally you don't expose the attributes directly anyway, you create methods for other objects to use.

 

So instead of having

Script X:

Property Form A
Property Int B

And then basically having stuff references if X.B == 3 then ...

 

You'd rather make

Script X:

Property Form A
Property Int B

function get_a
return self.A
endFunction

function get_b
return self.b
endFunction

function set_a ( a )
self.a = a
endFunction

function set_b (b)
self.b = b
endFunction

If the latter would still work, I think it's more like an improvement than limitation honestly. How would that work with CK though - I mean when you assign values for script properties directly from CK?

 

I guess I should take a look. >.>

[Gawad]

 

 

 

markdf over on LL is developing a tool that does pretty much exactly that: moves a papyrus script's properties to jarrays. He's looking at having it work like a Skyproccer. He did point out, however, that mods patched this way can't act as masters to any other mods, as those mods wouldn't be able to find their master's properties. So that would remain one important limitation.

 

Oh and about that 32-bit save breaker...euh, yeah, 32-bits worth of strings on a 32-bit limited program...that really was a brilliant suggestion, not. :huh:

 

 

Hmm... is that really a problem? In regards to quality of code and such, normally you don't expose the attributes directly anyway, you create methods for other objects to use.

 

If the latter would still work, I think it's more like an improvement than limitation honestly. How would that work with CK though - I mean when you assign values for script properties directly from CK?

 

I guess I should take a look. >.>

 

 

Sorry, I have no idea, he hasn't released anything yet as he is still putting together a prototype. The best thing to do would be to ask him directly; he's the authority on his own work, obviously, and I don't want to be putting any words in his mouth (writing any words from his keyboard?). Besides, this is way, way beyond any puttering around I've ever done with the CK or TES Edit.

[TerraKitsune]

 

 

Sorry, I have no idea, he hasn't released anything yet as he is still putting together a prototype. The best thing to do would be to ask him directly; he's the authority on his own work, obviously, and I don't want to be putting any words in his mouth (writing any words from his keyboard?). Besides, this is way, way beyond any puttering around I've ever done with the CK or TES Edit.

 

I figured, thus the 'should take a look' :) And it can wait until it gets a bit further along - I figure if it's in that kind of early stage, anything can still change anyway.

[cdcooley]

What about arrays? If you have say..

String mystr = "Hello world!"
String myweather = "It's raining again!"
int i = 0
int j = 10
int s = 2

That code would most likely only put 4 new strings in the table "mystr", "myweather", "Hellow world!" and "It's raining again!".

The i and s are already used in the game's own scripts and many modders use j, so even at the worst it's 5 strings if no one else used j yet.

 

String[ ] myStrings = new String[50]
Int[ ] myInts = new Int[50]
myStrings[1] = "Hello world!"
myStrings[2] = "It's raining again!"

That one's got four strings: "myInts", "myStrings", and the two string values.

 

 

Here's an example that only has two strings for the table (CDC_StringExampleScript and mySuperSpecialVariable) even though it uses strings. Anyone thinking of storing an array of strings should consider this option.

ScriptName CDC_StringExampleScript extends ReferenceAlias

int mySuperSpecialVariable = 0

string Function SpecialMessage(int MessageNum)
    if MessageNum == 0
        return "Move along, nothing to see here."
    elseif MessageNum == 1
        return "Where is everybody?"
    elseif MessageNum == 2
        return "Can we please leave now?"
    else
        return "I hate crowds."
    endif
EndFunction

Event OnActivate(ObjectReference akActionRef)
    Debug.Notification(SpecialMessage(mySuperSpecialVariable))
    if mySuperSpecialVariable < 3
        mySuperSpecialVariable += 1
    endif
EndEvent

That's not too much uglier than using a string array to hold the 4 string values. And if I had just one counter variable like that I would have just called it "i" which means the only string added by that script is the script name.

 

On a related note, it's always better to re-use existing script than to create a new unique script for a task. The developers created a lot of scripts starting with "default" that they use for various things but I've seen lots of people creating their own unique versions just so they can use a property name that will auto-fill in the CK. All of those unique scripts are adding at least two entries to the string table (the script name and the unique property name).

 

Even if you take a worst-case situation (of a game loading as many ESPs a technically possible) the average mod can still add over 130 strings. Obviously there will be some mods that add none or very few, so really the effort needs to be in trying to deal with the relatively small number of mods that add many hundreds of strings. Trying to push strings off into some external cache and writing completely unreadable code isn't really needed (at least not for the vast majority of mods).