LE Best tool for creating Data Structures in Skyrim Papyrus? (JContainers or alternative?)

[opusGlass]

PapyrsUtil is another skse plugin that works with large lists/arrays with 10000s of entries. Enderal uses it too.

 

Only load external data when the game loads and only save it when the game saves or otherwise manual. Meanwhile the data is in memory anyways, so using it is not hitting the disk.

 

If a mod requires another mod like PapyrusUtil or JCointainers, one problem are other mods that bundle these plugins. A version check in your scripts with an explanatory error message in case there is an outdated version installed works wonders in regards to users being able to help themselves and keeping support questions about that to a minimum.

 

PapyrusUtil beats the data size limit but doesn't allow data to be organized into structures as far as I know. Will be sure to include version checks, thanks!

 

 

Something that I've had a (little) experience with in FO4's Papyrus system is the use of multithreading. As an example I have my FO4 Lights Mod, which allows the player to shoot at physical light sources and the game will then disable any nearby light emitters.

 

The problem I ran into was that some cells have a large number of physical light sources / light emitters, upwards of 150+ depending upon the cell. Using a simple While loop to run through each physical light source and connect everything together took upwards of 10-20 seconds to complete. So what I ultimately ended up doing was creating 6 "activators" that would work on a chunk of the physical lights, each activator having the same script attached. This, obviously, sped things up considerably so that as pretty much as soon as the cell finished loading the light system was ready.

 

Now I created those activators statically (RenLightActivator01-06) but, and I'm still testing this out, I'm pretty sure it could be done dynamically. I honestly don't know how well something like the above would work for your needs, but I wanted to briefly explain how I approached a similar problem in the hope that it helps you with yours.

 

Thanks for the tip -- I probably won't be able to multithread this, but I'll keep that in mind if in case I do find a way!

Edited October 17, 2016 by opusGlass

[Reneer]

Oh, also, you might want to check out FO4 Papyrus' (and I assume Skyrim: SE) use of Structs, if you haven't already.

[Surilindur]

Now that really does sound big. Maybe if I have the time, I could see if I can think of something simple (all I really can think of). I had a sort of similar idea myself with external input that was supposed to delevel all leveled lists that were fed to it from a json file but I never really got around to making it happen, there was something in it that was an issue, and I cannot remember what it was. I was planning to use PapyrusUtil (JsonUtil from it) myself, but maybe JContainers has more elegant ways to handle things. Hmm. Might be worth looking into.

 

If people do not feel like downloading the utilities separately, then that is just lazy. I would also say that checking if the user has an appropriate version of the utility loaded should be enough, and then not calling the functions and maybe adding a notification if it is not (with a clear info on which mod the notification comes from and all that). Some people will always have something to complain about.

 

Hopefully you will find a useful solution. Waiting for a minute or two is a bit... well... I am not sure. It sounds like a long time, but with Papyrus being what it is, then maybe it cannot be helped. This actually sounds interesting, and maybe if I have the time, I could see if there is a simple way to make it faster (probably not). Maybe some small things in actual implementation could also make a difference. Rethinking all that does not really sound like much of an option, though. :confused:

[Surilindur]

So I have been playing around with PapyrusUtil, and it seems to maybe offer some interesting ways to achieve that. Maybe. While it does not seem to offer things like structs (I am not sure what would, unless they really do add those in SSE), it apparently does make it possible to create custom structured json files quite well - the commands are labeled "experimental" but seem to work just fine. For example generating an MCM page full of toggle options from a json worked like a charm, and only took less than a hundred lines of code script (the whole script). I need to play around more with this, write something that generates a few thousand items in a json and see how the performance is affected. Of course the actual adding of things to leveled lists and such probably take most of the time the script needs. I will report back when I have finished the whole thing with leveled lists and others. This is actually quite interesting.

 

If someone finds a working solution before I do, feel free to share, I would love to see it too. Chances are I will only manage to reproduce opusGlass' implementation and be stuck in the same position when everything has been added to the script. Only one way to find out. :tongue:

 

Edit: Also, does anyone know how PapyrusUtil/JsonUtil handles data? The comments say that the user does not need to worry about saving the file, as that would be done automatically. Does it keep it all in memory before writing it to a file? That would be great, but sounds a bit too good to be true. I wish it were so. There is this at the very least:

- When the player loads another save without saving themselves or the Save() function having been manually called by a script, the loaded data will be discarded and revert back to whatever the contents of the current saved file are.

 

Edit 2: Ah, so you need to check if the form already exists in the LeveledActor list right? Or not? Even just adding them one by one looks terrifying. Or maybe I am just doing it wrong. Nevermind, I want to see what happens anyway. This is very very interesting.

Edited October 17, 2016 by Contrathetix

[Surilindur]

So I finally have something that I can actually test for performance, but it is getting late and I will do it tomorrow. Storing all in a json file definitely works with PapyrusUtil/JsonUtil too, I have the following in it at the moment (for reference):

 

 

{
    "$_CX_JT_Page000" :
    [
        {
             "active" : 1,
             "default" : 0,
             "option" : "_000",
             "source" : "211369|Skyrim.esm",
             "static" : "3428|JsonTesting.esp",
             "target" : "124784|Skyrim.esm"
        }
    ]
}

And when loaded into a menu, turns into a page named "$_CX_JT_Page000" with a toggle option "$_CX_JT_Page000_000" that is disabled by default, but that has been toggled on by the user. And then the lists for adding things to and from, in the form PapyrusUtil stores them when using the Form commands for getting and setting them. The outcome is actually quite neat, page and option strings double as localisation keys. I did not check JContainers, it looked a bit too... sophisticated for my little brain. Maybe it would offer even better options.

 

I will try to copy-paste two thousand items into the thing and see how performance is affected tomorrow if I have the time. Chances are it actually will take minutes, though, but I really need to see it to believe, all the scripts I have written have been so small that I have no idea how slow Papyrus actually is. Also, the way I have done it (in a dumb way), it scales not-so-well with input size, so I think it will be interesting to see what happens. I will post/link my testing thing here when it is finished so people can laugh at it. :P

 

You mentioned having a FormList of references. Does that mean ObjectReferences? I have not really done much with FormLists, but it is apparently not too straightforward to add ObjectReferences to a FormList. Did you do it with a script, or in another manner? Or was it just an idea? Maybe I missed something, would not be the first time.

[opusGlass]

Hey thanks for looking into this! I haven't had a lot of time but I'm still plugging away at learning JContainers. Mostly writing test code to make sure everything is going to work as I expect, one snippet at a time. And I'm starting to think that parts of my plan were not going to work, so Structs might not be the best way to go anyhow.

 

You mentioned having a FormList of references. Does that mean ObjectReferences? I have not really done much with FormLists, but it is apparently not too straightforward to add ObjectReferences to a FormList. Did you do it with a script, or in another manner? Or was it just an idea? Maybe I missed something, would not be the first time.

 

My plan for the FormLists is this:

 

Users (mod authors) create a source leveled actor, LA_DragonsSource_1, which contains all of the entries that they want to add to the destination vanilla leveled actor, LA_DragonsDest_1.

- Users will create many of these. Often times, they will create more than one source per MCM option. (For example, Dragons have 7 vanilla leveled actors, so you would want 7 source LAs for the 7 destinations, all controlled by the "Dragons" MCM option.)

Users will then add all of the Sources for a single MCM option into one FormList in the CK (FL_DragonsSource), and all of the corresponding Destinations into another FormList (FL_DragonsDest).

Users can also place Actors and/or LeveledActors directly into Cells (through the usual means). If you place several dragons this way, you would create a FormList for these statically-placed dragons, FL_DragonsStatic.

 

One problem with storing the data from these is that merging mods and/or changing file names would likely cause any saved reference IDs to become invalidated. Additionally, I don't want authors to have to type out all of this info into a JSON file (or even worse, directly into the script).

 

 

In light of that, it seems like the best approach is for the authors to place all FL_*Source, FL_*Dest, and LA_*Static lists into respective Formlists: FL_Source_Master, FL_Dest_Master, FL_Static_Master. That way these can all be sent into a single script with only 1 property each. That leaves the Name and Default parameters. I think we can infer the Name value from one of the FormLists, and then the author could set up a JSON file containing a Map structure (don't know if PapyrusUtil has Maps, but JContainers does) to set the Default based on the Name. I could assume that all unspecified values are true to save the authors more time when creating that JSON.

 

One concern though: if the load order is changed or a mod is merged, the FormIDs might change. I believe FormLists will automagically correct all references in these cases. But will JContainer/PapyrusUtil structures break, even if they are stored in the save file rather than a separate JSON?

 

 

PS: I've been thinking about what you said re: parallelizing the problem. One alternative to the above would be to have a single script for every MCM option, i.e. a single script that adds dragons and another that adds bandits, etc. One problem is that I would need to find a way to communicate with each script from the MCM script, but I think that would be easy since JSON files can be read globally. A lot of the features that I have in DDC2 would create race conditions if they ran in parallel, but I think its good for me to remove those features anyway, since they will inevitably create conflicts between mods. Perhaps I will release a stand-alone mod that is designed to perform those functions on ALL leveled lists.

 

 

 

EDIT: Re-reading your question, it looks like you might be asking how to create a FormList of ObjectReferences in the CK. I think this should be straightforward, just drag them from the Cell View window (not the render window).

Edited October 19, 2016 by opusGlass

[Surilindur]

<snip>

 

So you had it imagined that way. Interesting, I imagined it a bit differently. Thinking about it, saving authors the trouble of writing it all out into a json file might actually be handy. Oh well. I will adjust my test thing, see how it works with a number of lists, and then post it here in case it will be of any use.

 

It was Reneer who mentioned parallel processing or such I think, not me (unless you meant "you" in plural). There was a small tutorial in the wiki on how to use multithreading in Skyrim Papyrus, with a summon spell example. At the time I read it, it was a bit too complicated for me. Which actually reminds me: I am still learning how everything works, so whatever I may come up with, it will probably be sub-par. Considering how mator also recommends JContainers, it will probably be better when it comes to performance.

 

Anyway, I will see if I can make something that works. It will probably just concentrate on making an MCM menu with all its data stored in json. The actual data would be stored in the json and not in the game at any time (except for when PapyrusUtil has it loaded in memory). It was the way I had it imagined, but it sounds like it will not be anywhere near enough. :blush:

 

I will try to make something sub-par, then post it here, and hope it helps. At least it should provide an idea or two. Or not. Hopefully someone else can help you with the more complex aspects.

[opusGlass]

> It was Reneer who mentioned parallel processing or such I think, not me

 

Oh yeah. Sorry about that!

 

> I will see if I can make something that works.

 

Anything you can do will be much appreciated! I'm looking forward to seeing your source code, it's always good to get an outside perspective.

[Surilindur]

So I have a sort of working version now (not quite, but enough for the loading part). It really does take minutes. I never thought Papyrus could be so slow. Well. The way I did it is actually really dumb, it just goes through everything and does nothing in parallel, and it uses lots of loops, so it will take forever obviously.

 

However what I did notice was that actual loading and saving of data from and to json seems to be fast, and also accessing that loaded data seems to be fast. Using Utility.GetCurrentRealTime, it said it took about 0.018 seconds to load one int + two forms from json, but I think it might actually be less than that when the game is capped at 60 frames per second and calling the functions to get the time also take time. Also, when the FormLists have been saved in a variable and all that, the actual modification of leveled lists does not use json or anything loaded from it, it uses FormLists and LeveledActors in the game itself. Or that is how I think it works, it would be odd if it somehow worked otherwise, but I am not an expert.

 

I think the actual issue is when looping through all the various lists inside lists inside lists and copying things. That happens inside the game, so I think that sort of solves it. Or not. Maybe another sort of design for the list system and such would work? And then using parallel processing for the actual list modifications somehow... ? To cut down the loop+copy part of it? So that there would be two FormLists for each json element, and both would hold a number of LeveledActors. The contents (each Actor) of each LeveledActor in source would be copied to each LeveledActor in target. It sounds a bit odd but at least it is very inefficient. Maybe I understood something wrong.

 

And here is the actual testing thing --> link. It is a bit *cough* unfinished *cough* and all that, being purely for testing, but if it helps, then that is great. :blush:

 

If there is something unclear about my test thing, I can see if I could explain it. There might be a pattern for the MCM option IDs, where first page is 256 forwards with even ones on the left and uneven ones on the right, and the second page also like that, but starting with 512. Or maybe not, it could just look like it. If there is a pattern, then the arrays for storing option IDs and json IDs could be gotten rid of, and the json IDs could be calculated from the MCM option IDs. Maybe.

 

Hopefully someone else can help you reduce the actual work that is done when copying things. I think that is the issue, and not the json part. Of course I have no idea what JContainers can do, so maybe it has a few tricks up its sleeve for processing things in-game when they have been loaded. And maybe it is an overall better option. I have no idea, I really should take the time one day to learn it.

 

I hope you will find a way to make your plans work while also being performant. Sorry for not being able to be of more help. At least this was interesting, and I finally managed to learn how JsonUtil works and all that. I think I mentioned it already, but I am still learning, so my skill level is a bit low. If any of that helps provide a new perspective, then I think it was worth it.

Edited November 26, 2019 by Contrathetix

[opusGlass]

 

...

 

Hey thanks man. It's good to know that the JSON interaction probably won't have a performance impact, and it's definitely been helpful to have you working on the problem with me. (That picture is spot-on by the way, that's exactly how I plan to do it.)

 

I think the only answer to the speed problem will be splitting each MCM option into a separate script so they can run in parallel. I could write the player's MCM preferences to a JSON file, then each of the separate option scripts can read these preferences and add their corresponding entries into the leveled lists. One problem with that though -- I don't know how to detect a game reboot outside of the MCM script. I could maybe detect the boot within MCM, then send that info to each of the respective option scripts, but I would somehow need to do this asynchronously (otherwise we'd be back where we started).