Custom cosave?

[xkkmEl]

I have a DB of sorts (> 22000 alchemy recipes) stored as a JContainer.

It turns out that holding this much data in RAM is causing major lag through load doors and on save/restore operations.

On disk, if I dump it to json format, it takes 9 megs.  I see that it also bloats the SKSE cosave by 9 megs.  Clearly this is what causes the lag... Although dumping to json on its own is easy and efficient (it causes no perceptible impact in game, no matter the load from other sources on the game engine), it really makes an impact in the load doors and save/restore (adds some 30-40s!).

The data is not required all the time.  Just when performing alchemy-related stuff (to use the recipe data), and when new ingredients are discovered (to add recipes using the new ingredient).  Keeping the data on disk the rest of the time is acceptable, and being struck with lag if you save/restore while the data is in RAM would also be acceptable since it would rarely happen.

The trouble is that the data is generated dynamically... it is dependent on your currently loaded mods that include new ingredients.  If I write it to a shared file, and you load alternately different saves, you would end up with inconsistent data.  If I write it to different files each time, I'll leave the user with many garbage unused files.

In short, I need something similar to the SKSE cosave... a file that will be deleted automatically when the save file is deleted.

I doubt I will find a viable solution for this... but before officially abandoning, I thought I should ask the wise ones on this board.

It could also be that the lag can be managed in other ways... I am thinking that perhaps it is not the volume of data as such that is responsible for the lag, but the number of Form references in the DB.  If that's the case, I could change how the DB is encoded to minimize the use of Forms (use an index into a consolidated list of Forms... so that each ingredient Form is referenced once-only instead of once-per-recipie-using-that-ingredient).  If that is the case, my lag would be reduced at least 25-fold and my problem would be solved.

 

[xkkmEl]

On 2/8/2024 at 5:56 PM, xkkmEl said:

It could also be that the lag can be managed in other ways... I am thinking that perhaps it is not the volume of data as such that is responsible for the lag, but the number of Form references in the DB.  If that's the case, I could change how the DB is encoded to minimize the use of Forms (use an index into a consolidated list of Forms... so that each ingredient Form is referenced once-only instead of once-per-recipie-using-that-ingredient).  If that is the case, my lag would be reduced at least 25-fold and my problem would be solved.

That hunch turned out right.

I did that, and I converted my 22K JMaps to JArrays using a positional representation.  The size of the json was divided by better than 3; the SKSE cosave by 2; and fast travel/save/restore time is no longer an issue.