Corrupt Saves - StrCount > 0xFFFF - CTD on load

[DarkRisingForce]

Sadly I have only recently had this problem

 

Here's my last loadable save (autosave3) lvl 25 character (27mb file size)

 

http://i68.tinypic.com/1yrg42.jpg

 

I have noticed that most of the script instances are DynDoLod

 

Can't find the culprit though among my modlist

 

Small update:

 

After deactivating DynDolod

 

http://i63.tinypic.com/2i0ykbq.jpg

 

Strings down from 65k to ~45k

And script instances from ~232k to ~107k :ohmy:

Papyrus size from ~22 mb to ~10.2 mb

Never knew it was this heavy

[bcsp]

Ouch. That's a hefty bunch of strings.

We were confused about dyndolod. Someone measured it over on LL and said it was really low. Seems dyndolod is one of those mods that starts low and grows.

 

I haven't had a chance myself to try out the beta scripts.

[DavidJCobb]

Mods with expansive complex MCMs. Each MCM entry (slider, toggle, whatever) requires several variable definitions. Scrolling through the strCount table I see lots of variables with 'OID' in their name and they're just the variables from mod authors that use that convention. I sigh when I see mods adding pointless MCMs. Keep it simple folks.

I'm curious as to whether state options alleviate this problem. Old-style MCMenus require tons of variables to cache option IDs, but state options use a much more elegant syntax: you define UI event handlers in different script states, with one state per option. If states don't bloat the string table just by existing, then state options may be reasonably safe (aside from being more maintainable and less hideous anyway).

 

Someone should write a test script with hundreds or thousands of states, and then see if these state names bloat the string table if any function in the test script runs. I might set that up myself (with other tests) if I have the time to do it today.

 

EDIT: SkyUI maintains its own string table in Papyrus, matching state names to option IDs, so actually this could still be an issue... :\

Edited March 26, 2016 by DavidJCobb

[bcsp]

Someone on the other forum suggested using json for mcms posiibly. I don't exactly know what they are so I'll leave it to more clever people to explain. Edited March 26, 2016 by bcsp

[bcsp]

Jcontainers and papyrusutil (we're hoping) has been successfully used to reduce strcount. Not in mcms though. Not yet at least. Edited March 26, 2016 by bcsp

[registrator2000]

Thanks for sharing this. Would be interesting to find out if the same issue exists with Fallout 4. Seeing how little has changed, I'm inclined to think it will!

[DavidJCobb]

An SKSE-based API like the following could make it possible to store large amounts of values outside of Papyrus, such that none of them have to be keyed to a variable name. However, that wouldn't help out with properties, which are probably the largest drain on the string table. I'm not sure it'd be terribly useful at all, so think of it more as a thought experiment.

 

Off-the-cuff unchecked draft:

 

C++-side code:

 

template <typename T>
class PerPluginDataStore<T> {
   class PluginEntry {
      class DataEntry {
         UInt32     identifier;
         DataEntry* nextEntry;
         T          data;
      }
      DataEntry*   firstEntry;
      PluginEntry* nextPlugin;
   }
   PluginEntry* firstPlugin;
}


PerPluginDataStore<char*> map; // for strings

You could then set up a Papyrus API like this:

 

Scriptname ArgleBlargle extends Quest

Int Function GetPluginID(Form akForm)
   Return Math.LogicalShift(Math.LogicalAnd(akForm.GetFormID(), 0xFF000000), 0x18)
EndFunction

Function SaveStrings()
   SKSEPerPluginDataStore.Save(GetPluginID(Self), 10, "Unique String")
   ;
   ; plugin ID, numeric identifier within the data set, string to save
   ;
EndFunction

;
; Assuming proper cleanup, "Unique String" shouldn't be in the string table unless either of 
; these functions is currently running. But then, nobody's verified proper cleanup yet. I'll 
; have to get around to that.
;

Function LoadStrings()
   String sTest = SKSEPerPluginDataStore.Load(GetPluginID(Self), 10)
   ;
   ; plugin ID, numeric identifier within the data set
   ;
   ; above call retrieves "Unique String"
   ;
EndFunction

 

 

Properties are a trickier case. When they refer to "meta" assets (e.g. Message, GlobalVariable, EffectShader), you could replace them with calls to GetFormEx or GetFormFromFile. Your code will be hideous, substantially less maintainable, and very marginally slower at run-time, but you'd save on property names.

 

Int iTemporary = (GetFormFromFile(0x00EEEEEE, "MyPlugin.esp") as Message).Show()

That's probably only needed as an emergency fix, for mods that have attempted all other options and still have too many property names.

 

----------

 

Interesting NPCs is noted for having tons of properties. A cursory examination of their dialogue fragments finds several scripts that are functionally identical but have varying property names. One example: there are dialogue options to dismiss a companion to different player houses: these scripts are identical save for the names of their properties (e.g. PaleHouse, FalkreathHouse, and so on); these scripts could have their properties renamed to pkHouse, collapsing some nine properties into one. In fact, were it not for the Creation Kit's disastrously awful dialogue-fragment system, these dialogue options could even use the same script (and with sufficiently skillful tampering in TES5Edit, one could perhaps accomplish that). See for yourselves:

 

 

 

;BEGIN FRAGMENT CODE - Do not edit anything between this and the end comment
;NEXT FRAGMENT INDEX 1
Scriptname TIF__0221A177 Extends TopicInfo Hidden

;BEGIN FRAGMENT Fragment_0
Function Fragment_0(ObjectReference akSpeakerRef)
Actor akSpeaker = akSpeakerRef as Actor
;BEGIN CODE
GetOwningQuest().SetStage(10)
(GetOwningQuest() as RelationshipMarriageSpouseHouseScript).MoveSpouse(akspeaker, PaleHouse)
;END CODE
EndFunction
;END FRAGMENT

;END FRAGMENT CODE - Do not edit anything between this and the begin comment

Referencealias property palehouse auto

;BEGIN FRAGMENT CODE - Do not edit anything between this and the end comment
;NEXT FRAGMENT INDEX 1
Scriptname TIF__0221A176 Extends TopicInfo Hidden

;BEGIN FRAGMENT Fragment_0
Function Fragment_0(ObjectReference akSpeakerRef)
Actor akSpeaker = akSpeakerRef as Actor
;BEGIN CODE
GetOwningQuest().SetStage(10)
(GetOwningQuest() as RelationshipMarriageSpouseHouseScript).MoveSpouse(akspeaker, FalkreathHouse)
;END CODE
EndFunction
;END FRAGMENT

;END FRAGMENT CODE - Do not edit anything between this and the begin comment

ReferenceAlias Property FalkreathHouse  Auto  

 

 

Similarly, I'm seeing dialogue fragment scripts that don't do anything other than call SomeSceneProperty.Stop() and in each of these, the scene property has a unique name (e.g. ErevanScene1). These, too, could be renamed (to pkScene or something similar) and doing so would fold all such properties into a single entry in the string table.

 

Changes like these could substantially reduce Interesting NPC's impact on the string table. I believe the primary concern in making these changes would be to use TES5Edit to prune the old property data off of the dialogue forms; failure to do so would result in a Papyrus warning (on savegame load) for every renamed property. I have a TES5Edit script that can strip invalid property data (left behind after a rename) off of forms; I have another that can rename a property defined on a form (e.g. you can edit and recompile the Papyrus source, and then use the TES5Edit script to update property definitions on forms). I'm happy to provide these if anyone needs them.

Edited March 27, 2016 by DavidJCobb

[bcsp]

Hindsight is 20-20.

Personally I've been seeing this issue for a few years now. Only lately do we know what actually causes it. We can't really fault interesting NPCs for it. Nobody knew about it.

I wonder how Skywind is faring on this front. Are they gutting skyrim properties to "make room". My guess is that they are not aware like 99% of the Skyrim community.

 

Yea, duplicate property names don't get added to the string table but the problem is that we modders are a lazy bunch :D

Nobody is going to want to rewrite their scripts to suit a minority. And interesting NPCs has over 6000 (I think) fragments. Would it be possible to automate the process of renaming properties reliably with a phyton script or something similiar?

 

Edit: There's also the question of script readability. Renaming FalkreathHouse to "Property1" and palehouse to "Property1" does not make for easily understandable scripts. This is where the idea of a precompiler comes in. Rename properties on the fly and send them to the papyrus compiler. Nobody cares about the readability of their pex files.

 

Of course the simplest solution is to simply make strcount at 32 bit field but that takes very specialized knowledge.

Edited March 27, 2016 by bcsp

[DavidJCobb]

I certainly had no intention of highlighting their work as bad practice.

 

Names like Property1 aren't legible, but names like pkHouse (for the house that a given line of dialogue sends a marriageable NPC to) are self-evident.

 

I'd advise a workflow like this for fixing Interesting NPC's problem:

 

Start by opening all of the scripts with properties to rename, in Notepad++. I'm pretty sure it has a feature to find-and-replace in all open files, and you can set it up to run the Papyrus Compiler via command line; ergo, use find-and-replace to rename the properties, save all the updated source files, and then compile the scripts one-by-one, each right after the other.

 

Open the mod itself in TES5Edit. Select all forms with renamed properties, and use my TES5Edit "rename property" script to update the property data bound to these forms -- one pass per property. If you rename nine properties, you just run the script nine times.

 

And then test the updated mod with an old save to make absolutely sure that Skyrim doesn't whine at you about the renamed properties despite your hard work.

 

So I just gotta get a Github account and upload my TES5Edit scripts there, and all the needed tools will be available. The author will be able to identify scripts that can share a property name, and update them in batches.

[bcsp]

Ok, that sounds interesting. Let me know when you get set up.