Reducing data saved to save file / memory usage.

[csbx]

My mod uses a substantial number of very large String Arrays (20+), each has an avg of 30 entries, each entry is ~250 chars.

I've set things up so that I've stored all of this to a txt file and then read via papyrusutil, the entire file, temporarily storing this in string arrays (lazy-loaded); and then when I'm done using them, I attempt to clear all of them via ARRAYNAME = none across the 20ish arrays.

I haven't until now thought much about whether this system is functioning properly--my question now: does this look like it should ? 

My main focus, obviously, is limiting what oozes into the save file and freeing memory in general, but I don't see a lot of documentation on the matter.

 

[xkkmEl]

I have been abusing JContainers to store large databases.  I have a DB with >22000 alchemy recipies... something like 4 megs of data.  No issues, almost.

1st, your data of ~600 strings of ~250 chars will take just 75K or so.  NOTHING to worry about.

2nd, what did make a big difference in my case, was minimizing the number of forms referenced in my db.  The save and restore processes still have to contend with the game's lock design (mutual exclusion locks), which really really slows things down.  So instead of referring to an ingredient in 1500 recipes, I had to make an ingredient list and store my recipes using indices into the ingredient list.

My recommendation is to keep it simple.  Don't optimize until you see an issue.  Your lists of strings are quite unlikely to cause anyone any trouble.  You'll be competing for memory with 60M texture files; 75K more or less won't register.

[PeterMartyr]

@csbx there a reason data get baked get's baked into the game save and it is over all performance, it is how the game optimises it self, you would be hamstring (lol pun intended) your mods performance if you did that. It is only an issue for Gamers that add or remove mods in mid-play-thru, That has never been recommended to do as well, basically two wrongs do not make a right, your trying to appease Gamer that we call the loud ignorant bunch.

Papyrus does have an auto read only option, but I will add this, papyrus is buggy concerning the ASCII or ANSI table, to the degree that the SKSE team has DISCLAIMER over it, in String Utility.  You can try to store your strings in that... but I am not recommended it, I have seen that bug first hand and it heart breaking.

Go with PU or JC for that, but you cannot write code that is not hardcoded without using a const or a read only data type, and that is how you optimiser your memory usage, since strings variables or returns,  in Papyrus are not case sensitive, the string bug has no effect there and most will not even know it is there.

But for string data that display to the screen, I would not use it

<Flags>; there are four flags available for you to use on your properties: Auto, AutoReadOnly, Hidden and Conditional.

Oh the TESV_Papyrus_Flags.flg is just a list of keywords, or it try's to be, open it in Sublime, it is very over rated))) it just read only (pun intended )instruction for the compiler))

  Quote

AutoReadOnly properties are similar to Auto properties in that, again, Papyrus will do the nasty background stuff for you, but with a couple extras:

 

1. You must provide a default value inside the script by filling in the <Optional Value> parameter, you cannot supply a value through the Creation Kit. Consequently, this means you cannot declare properties that point to game objects as AutoReadOnly, only Variable properties.

2. The property is read-only, meaning its value cannot be changed ever (with one exception).

3. That one exception: normally once the game loads a property, its value is permanently stored in your save. AutoReadOnly properties, however, do not get saved.

Expand  

https://tesalliance.org/forums/index.php?/topic/5039-class-2-properties/

[PeterMartyr]

Basically storing Constance Data in the RAM, (and the game save) no matter what language you are using, is a (Crood Movie) bad bad bad bad bad bad bad thing, that is not how we roll.... Good to see you are thinking about it

 

[PeterMartyr]

BTW Came back for this, if you do this

String text = JsonUtil.GetStringValue("FileName", "KeyName")
Debug.Notification(text)

what have you achieved? You declared a variable and are using RAM. So nothing.  Plus it is hard coded.  This is how to not use RAM

Debug.Notification(JsonUtil.GetStringValue(FILE_NAME, KEY_NAME))

So it will need Const/AutoReadOnly to back it up for the FileName and KeyName. Plus it is now soft coded 

 

Is it worth it to you?

btw with option  A and the local variable OFC it will not get baked into the game save, but it only fulfill 50% of the post and runs slower than option B and need the windows garbage disposal unit to clean it up... FYI free unused RAM once it is no longer required.