LE Best tool for creating Data Structures in Skyrim Papyrus? (JContainers or alternative?)

[opusGlass]

I'm currently learning to use JContainers by silvericed, but before I dive too deep I'd like to see if there are any other options.

 

One concern is that users must install JContainers separately (he advises against bundling JContainers with your mod to prevent version conflicts). I'd like a resource that can be bundled into my mod without conflict if such a thing exists.

 

Another concern is that this will be slow when scaled up to contain large amounts of data -- my script already takes up a lot of processing time (a couple minutes to run) and there is a large amount of data that needs to be stored. Since JContainers stores the data in separate files outside of Skyrim's engine, I'm worried that reading and writing might slow my script even further.

 

Do you guys have any other recommendations?

[dAb]

Formlists of formlists is the closest thing you can have to a multidimensional array, if that's what you're looking for.

[Surilindur]

How much is a large amount of data, what sort of data, and how are you planning to store it? Just curious, on a general level. You make it sound like it really will be huge, and I am curious as to what it is you are going to be storing that actually is huge in Skyrim scripting.

 

Maybe the idea could be rethought somehow, to avoid having to handle lots of data? Of course it might not be possible but still. That does not mean I could help, but I just cannot help being curious. :P

[opusGlass]

Formlists of formlists is the closest thing you can have to a multidimensional array, if that's what you're looking for.

 

I'm looking for something more along the lines of a Struct or a Class -- a custom-made object type for organizing data. They've added this in FO4 but don't have them in Skyrim.

[matortheeternal]

Pretty sure JContainers is your best bet, and I'm willing to wager it's more performant than any other solution you could hack together. (not that I doubt your ability so much as that it's pretty performant in how it's implemented. if you need to load a lot of data you should just pack it all into a single json file and load that file once when the game starts)

 

And the advice on not bundling the framework is good and should apply to all frameworks. Don't bundle frameworks. Your users should be plenty capable of downloading JContainers, and if you're really concerned about it being too hard you should build a mod dependency/bundling tool. OR at least check to see if JContainers is installed before running your script and produce a big fat error message if it isn't.

[opusGlass]

How much is a large amount of data, what sort of data, and how are you planning to store it? Just curious, on a general level. You make it sound like it really will be huge, and I am curious as to what it is you are going to be storing that actually is huge in Skyrim scripting.

 

Maybe the idea could be rethought somehow, to avoid having to handle lots of data? Of course it might not be possible but still. That does not mean I could help, but I just cannot help being curious. :tongue:

 

It's for a LeveledActor manager. LeveledActors aren't saved between loads, so any script-based changes need to be remade at every boot. And trying to make them customizable through an MCM-menu complicates things further.

 

I'll use my mod Diverse Dragons Collection 2.0 as an example of the complexity. Off the top of my head I think there were about 140 separate Actors that needed to be placed into 7 LeveledActor lists. Multiply this by ~1800 lines of code. So every time the game booted, it would take a couple of minutes to build your leveled lists if you have all dragons enabled. Slower computers were much worse.

 

For this "Manager" mod, one data structure that I'd like to create is the following:

 

 

 

Data associated with a single enable/disable option (for example, Fire Dragons: Enable/Disable)

Option name, a string

Enabled by default?, bool

List of statically-placed references, FormList

Source lists, an array/formlist of n LeveledActors

Destination lists, an array/formlist of n LeveledActors

Currently enabled?, bool

MCM option index, int

 

The reason I'd like to package all of these into a single object, is because there will be *tons* of them. It will become tedious to the point of impossibility if I just let these variables lie around loosely. For example, if I want to pass a hundred of the above structures as parameters to a function.

Edited October 16, 2016 by opusGlass

[opusGlass]

Pretty sure JContainers is your best bet, and I'm willing to wager it's more performant than any other solution you could hack together. (not that I doubt your ability so much as that it's pretty performant in how it's implemented. if you need to load a lot of data you should just pack it all into a single json file and load that file once when the game starts)

 

And the advice on not bundling the framework is good and should apply to all frameworks. Don't bundle frameworks. Your users should be plenty capable of downloading JContainers, and if you're really concerned about it being too hard you should build a mod dependency/bundling tool. OR at least check to see if JContainers is installed before running your script and produce a big fat error message if it isn't.

 

Thanks for the advice -- yeah I will likely end up sticking with JContainers. It's true that they can install it, but it'll be the last nail in the coffin for Steam Workshop with me (I usually post my mods to Nexus and SW). SW users have already been raising such a ruckus about the SKSE/SkyUI requirements on my latest mods. Bunch of ingrates honestly. (And even the ones who aren't toxic might still pass up on the mod just because it has a requirement that they're unfamiliar with.)

[matortheeternal]

Thanks for the advice -- yeah I will likely end up sticking with JContainers. It's true that they can install it, but it'll be the last nail in the coffin for Steam Workshop with me (I usually post my mods to Nexus and SW). SW users have already been raising such a ruckus about the SKSE/SkyUI requirements on my latest mods. Bunch of ingrates honestly. (And even the ones who aren't toxic might still pass up on the mod just because it has a requirement that they're unfamiliar with.)

 

You know what they say, "You can't fix stupid." :P

[Reneer]

Something that I've had a (little) experience with in FO4's Papyrus system is the use of multithreading. As an example I have my FO4 Lights Mod, which allows the player to shoot at physical light sources and the game will then disable any nearby light emitters.

 

The problem I ran into was that some cells have a large number of physical light sources / light emitters, upwards of 150+ depending upon the cell. Using a simple While loop to run through each physical light source and connect everything together took upwards of 10-20 seconds to complete. So what I ultimately ended up doing was creating 6 "activators" that would work on a chunk of the physical lights, each activator having the same script attached. This, obviously, sped things up considerably so that as pretty much as soon as the cell finished loading the light system was ready.

 

Now I created those activators statically (RenLightActivator01-06) but, and I'm still testing this out, I'm pretty sure it could be done dynamically. I honestly don't know how well something like the above would work for your needs, but I wanted to briefly explain how I approached a similar problem in the hope that it helps you with yours.

Edited October 16, 2016 by Reneer

[sheson]

PapyrsUtil is another skse plugin that works with large lists/arrays with 10000s of entries. Enderal uses it too.

 

Only load external data when the game loads and only save it when the game saves or otherwise manual. Meanwhile the data is in memory anyways, so using it is not hitting the disk.

 

If a mod requires another mod like PapyrusUtil or JCointainers, one problem are other mods that bundle these plugins. A version check in your scripts with an explanatory error message in case there is an outdated version installed works wonders in regards to users being able to help themselves and keeping support questions about that to a minimum.

Edited October 16, 2016 by sheson