Fallout 4 Optimization and Performance Systems Explained

[SMB92]

Ah come on don't shoot me down now :P Is it still possible to have hope for a Beth game?

 

I came to the conclusion that you can't have multiple generations in an area or close by, finding too much problems with the vis/occlusion bugs/old meshes. I combine all my mods at once, I just switch on the esm flag and load em all up and have the one gen for the lot, even if it means I need to gen 11 ugrids worth to catch everything optimally (cambridge good example). I wanted to start a patch project like this but time become an enemy.

 

My use for the script would have been to clean up the unnecessary files from these big gens rather than make multiple patches work together, i think thats too messy.

[VIitS]

 

I'm actually a bit confused about the script myself even though I'm the one who decoded the file side of it.. :tongue: If you remove unnecessary _OC files how is file resolution done? Does it go backwards through your load order looking at every <Plugin.ESP>\Precombined\<blah>_OC.nif looking for the matching name? Or does it simply jump all the way down to vanilla? If it's the former then I foresee potential issues. The filename in the lower priority mod could match but not actually have the same content as vanilla. I don't know if this can happen via generation (same name but different content) but that doesn't preclude someone from manually editing their precombined files. One can manually tweak the positions of things in precombined files for example.

 

Anyway if that fallthrough example is possible--basically having _OC files from 2+ different mods in the same cell--then what happens to the UVD files that only know about the state of the cell from when the top-most mod baked it? You likely only have the masters + your plugin loaded when baking, so the UVD really only knows about vanilla + your mod.

That is something I haven't figured out yet. I know loose files will be loaded even without a plugin, but I haven't gone through the process of making multiple mods for the same cell, generating precombineds, pruning the excess precombineds, packing them into .ba2 files, and then seeing how they all play together. I plan to at some point, but even with finals done I have a lot of stuff going on, so it'll probably be some time before I test that part.

 

The main driver for this script was me attempting to clean up SE's Mechanist's Lair and Vault 88 a bit. I realized that the file size could be massively reduced when I saw what precombineds the DLC had for the given cells, and the other part of the script (which few are likely to use, involving editing which references are "assigned" to which precombined meshes) was from me looking into editing the vanilla precombineds after someone pointed out the differences between them (I initially thought they might not be working right, realized that wasn't the case eventually).

 

Even ignoring the issue with previs, you also have the issue of the CELLID_physics.nif, which is generated with the rest of the precombineds, and contains the collision data (and possibly lights and such, not sure) for all the precombined meshes in the cell. Which means compatibility between two mods that changes things enough that both require an updated .uvd file will also both require a new _physics.nif. So you'd still need a compatibility patch made for it.

 

The good news about this information (and this script), is that something like SMIM is possible without killing performance. The visibility and collision would be unaffected, so you'd just need to generate new precombineds for the relevant mesh changes. Then delete the _physics nifs (which should be identical anyway), clean the excess precombineds (so the mod isn't 30ish GB or more) and just leave them loose so they don't require tons of cell edits that are just changes to the PCMB field*.

 

And the good news about adding precombined meshes is that it should be possible to have the collision data in the precombined mesh itself, based on one of the precombineds from NukaWorld (00002DF7_0971F8CA_OC.nif), which has cliff and stair collision data in it. That would probably require manually adding it to the nif though, so it might not be worth any performance gain.

 

*haven't actually tested if an updated PCMB field is needed to load packed precombined meshes, but if it isn't I'll be surprised

 

@Giggly, yeah, I've seen a few meshes that were unnecessarily complicated (such as the perfectly flat walls with far more vertices/triangles than needed to perfectly define the shape). And of course, fixing them would require you to generate new precombineds for them after you fix the mesh.

[SMB92]

So I guess the next thing to invent is an auto generator of collisions and physics for these precom meshes. What god among men can deliver that. Ah mind runs wild in troubleshoot mode...

 

Patches/combos it is then. I've got most of Beantowns done (one generation contains overlapping 5, 7, 11 grid generation for all Cambridge all the way to Bunker Hill, I find the performance is smoother myself, something was wrong in that area in vanilla I believe).

[hexabit]

@VIitS

 

Yeah, I brought up XPRI and _Physics in the *Edit thread on AFKmods but I forgot to carry that over here.

 

I've seen the HavokRoot node in tons of _OC files. I think a good rule is that if an _OC file (vanilla format) is >10KB there is a good chance that it has a HavokRoot.

 

I even thought that the physics from precombined references had to be in the _OC and that the _Physics was for other things because of the prevalence of HavokRoot. And after what you said now I'm not so sure. I originally thought that the XPRI list was what goes into _Physics, but it does seem like the lists for those are rather paltry. I have viewed the _Physics in CK Preview before with the collision wireframe and it did seem to be a lot more than the XPRI lists which is generally activators, FX and such.

 

What I do notice is that it seems like what is in XCRI is never in XPRI and vice versa, so I guess maybe the official way of saying what is in _Physics would be (XCRI + XPRI - the HavokRoots from the _OC files). I can see no real rhyme or reason for there being a HavokRoot instead of it going into the _Physics file however. I see a lot of trees and stairs in the HavokRoots but a lot of random objects too.

 

Have you discovered the method to the number of _OC files that you get out of a cell? There is obvious spatial partitioning going on so as to keep the meshes that get combined close together.. otherwise the bounding sphere that contains them can be as big as or larger than the cell. But then every cell should be 16 or 64 _OC files. Instead it seems like the average is closer to 32. Maybe 2x4x4 to make sure things are also vertically close? Basically, other than such a spatial partitioning scheme, is there any reason for the separate files that you are aware of? I also wish I was able to figure out the reason for the naming in the second section of the filename. Not to mention the hashes in the ExtraData block that actually point to the meshes.

 

---

 

...As for SMIM I still think it's in the category of "Virtually Impossible" while keeping the precomb system on. I've long professed this by saying that it necessitates regenerating the precombined for every cell in the game, thus including 2GB+ of baked data. What I didn't know then was that the files we can generate are 10x larger than vanilla. It's very unlikely given the scope of an SMIM-like project that 90% of the _OC files would be unnecessary to package (20GB->2GB) so it still ends up worse than my original claims. Let's be hopeful and say that you can get the 20GB of data down to 4GB. Even before adding in the actual NIF files that you are replacing, that's one of the largest mods ever. Not to mention the sweeping incompatibilities this causes with anything else needing regenerated previs.

 

I'm not even sure that the CK can regenerate the precombined for all affected cells when you replace an extremely common NIF (say a garbage pile NIF that has 7000 uses across nearly every cell). In the past the CK typically just crashes when given that kind of workload. :smile:

 

Also each time you change a single vertex or poly on an SMIM NIF, you may need to regenerate all 20GB+ of the precombined**, depending on the prevalence of the NIF in ref usage. For example if you tweak a NIF with just a few uses, you only need to regenerate precombined for a few cells. You tweak a NIF used in very cell, you need to regenerate the whole game. (**: Both versions of the data block in _OC files are directly reliant on the vertex and triangle counts in addition to the LOD level information.)

 

Given this, I think it would be much simpler to programmatically uncombine all references that use said super-common NIF with something like your script. I know that your script can uncombine a reference manually, but can it uncombine all references across all cells which use a certain NIF (or certain STAT, etc)? That would be extremely useful in this case. I guess the only downside is possibly doubled-up physics? SMIM-like mods normally don't change the collision shape anyway and you would only uncombine the references which have thousands of uses per NIF, so I don't really see the big deal of having the physics loaded from the precomb and the uncombined NIF given it dramatically cuts down on how many affected cells have to be regenerated when you edit said NIFs.

 

Taking that into account, you're basically creating a blacklist for certain NIFs and any refs that use the blacklisted NIFs should somehow be considered totally vanilla when baking precombined. I forget the exact wording in the CK menus, but it's something like "regenerate precombined for all affected cells" and I'm assuming this looks at any loose files you have and determines which cells this overridden NIF is used in. In that case I guess you can just move the NIF files out of the Data folder that you plan on uncombining. My reasoning for bothering doing this is also filesize. An SMIM-ified NIF file is probably going to be way bigger, and if the CK is sticking that data into the _OC files that means that they'd be a lot smaller because they'd still be vanilla.

 

So to summarize my ideas:

• The more super-commonly used NIFs that you uncombine (and keep out of the Data folder when baking) the:
  1. Smaller the total size of the regenerated _OC will be given that SMIM-ified files will be way larger (and that size differences gets multiplied by use count)
  2. Fewer cells which have any non-vanilla NIFs in them needing to regenerate, meaning less time regenerating it all.
  3. Fewer times you have to regenerate precombined at all when editing a file's geometry. If it's a permanently uncombined file, you do not need to regenerate the precombined to get it working correctly in game.

 

The only issue I can think of at the moment is that of initial setup. There may be a lot of cells where you only have one SMIM NIF at the time, and if it's an "uncombined/blacklisted" NIF you still need the cell record in your plugin to uncombine it.

 

Also I'm not certain just moving the files out of Data works for the way the CK determines what cells are "affected" when you regen precomb in that way. I should really go into the CK sometime as it's been so long I forget the exact terminology. :tongue:

[VIitS]

I've seen individual _OC files with 1000 references (the 00000000_OC.nif in the 0000E05A Cell, it had about 50 less meshes and 1000+ more references than a neighboring cell, 0000E059), and I've seen ones with a single reference. When shadowslasher regenerated the precombineds for the Mechanist's Lair (to allow more things to be scrapped, without letting you scrap outer walls), there were some references that were shuffled between _OC files (though that might be due to things that were removed from the precombineds, it's hard to tell). I don't have a clue how it figures out what goes in what mesh.

 

Yeah, just uncombining those would be much better. If you wanted to get rid of the duplicate collisions, you could delete the base records of the ones you want excluded in the CK, generate the precombineds, then only keep the physics nifs (and the ones with less-commonly used SMIM nifs). From what I have seen, the physics nif generated by the CK (for an identical cell) should be the same size. The one shadow made for the Lair was a few hundred KB smaller, and removed about 450 references from being precombined. The problem is, of course, that would make it incompatible with just about any mod that touches a cell (unless the mod disables precombineds like the scrapping mods do, in which case it'd be the same as it was in Skyrim), since uncombining requires cell edits. The only way you could do it without massive incompatibility issues is use a script (almost certainly f4se) to dynamically uncombine those references, which I'm guessing would require someone figuring out how to identify a reference (in the massive list of references the cell record contains) by the base record used for that reference.

 

Unfortunately, the script can only do one cell at a time. I'd have to check to see if it'd be possible to change that, but a SMIM-type mod is pretty much the only case I can see it being important. And there are so many problems (and so much work that would be required after the "improved" meshes are made) I don't see anyone bothering. So you're right, a SMIM mod would still be virtually impossible.

 

And yeah, considering how much I've learned about precombineds/previs (and how much I seem to be considered an expert on it), I've spent very little time in the CK myself :laugh:.

 

Edit: also, one note for anyone looking at the cell info in xEdit to figure out how many references are precombined: the "reference count" it gives is double the actual number

[zilav]

Edit: also, one note for anyone looking at the cell info in xEdit to figure out how many references are precombined: the "reference count" it gives is double the actual number

For some weird reason Bethesda count not the pairs of mesh and ref, but each one individually. It is a linear list even though xEdit displays it as an array of pairs for convenience.

[Giggilyomeromicon]

 

@Giggly, yeah, I've seen a few meshes that were unnecessarily complicated (such as the perfectly flat walls with far more vertices/triangles than needed to perfectly define the shape). And of course, fixing them would require you to generate new precombineds for them after you fix the mesh.

 

 

That's certainly a downside, yeah, but from my experience these meshes are only posing a serious problem in a few cells. Since the occlusion culling only works properly when fairly close to an object you wind up with scenes in downtown Boston with hundreds of these needlessly complex meshes being used. I'm hoping that just using them in a few cells will be all that's necessary.

[VIitS]

 

Edit: also, one note for anyone looking at the cell info in xEdit to figure out how many references are precombined: the "reference count" it gives is double the actual number

For some weird reason Bethesda count not the pairs of mesh and ref, but each one individually. It is a linear list even though xEdit displays it as an array of pairs for convenience.

 

I figured as much, every time I've had an issue with something in xEdit it was forced to be that way due to something Bethesda did in the record.

 

 

 

 

@Giggly, yeah, I've seen a few meshes that were unnecessarily complicated (such as the perfectly flat walls with far more vertices/triangles than needed to perfectly define the shape). And of course, fixing them would require you to generate new precombineds for them after you fix the mesh.

 

 

That's certainly a downside, yeah, but from my experience these meshes are only posing a serious problem in a few cells. Since the occlusion culling only works properly when fairly close to an object you wind up with scenes in downtown Boston with hundreds of these needlessly complex meshes being used. I'm hoping that just using them in a few cells will be all that's necessary.

 

If any of those cells have a (full sized) Red Rocket, it's a pretty big culprit for that sort of thing (particularly the roof/ceiling and floor). I cut the size of the Workshop_02 part in half, and I only messed with the parts where it was easy to fix the UV. Admittedly, some of that was removing a small detail that was fully blocked due to my "repair" of the mesh (it's still mostly blocked in the regular mesh, and not very noticeable in any case). Let me know if you want it.

[Giggilyomeromicon]

 

If any of those cells have a (full sized) Red Rocket, it's a pretty big culprit for that sort of thing (particularly the roof/ceiling and floor). I cut the size of the Workshop_02 part in half, and I only messed with the parts where it was easy to fix the UV. Admittedly, some of that was removing a small detail that was fully blocked due to my "repair" of the mesh (it's still mostly blocked in the regular mesh, and not very noticeable in any case). Let me know if you want it.

 

 

There aren't any Red Rockets around the spots that I'm editing, but I may take you up on that later.

 

I'm also having pretty good success with the mesh edits that I'm making. Only gaining around 5-10 in some of the major problem areas at the moment, but that's the difference between 20 and 30 FPS for me. Once I'm done just doing .nif replacements I'm planning on replacing more 3D models with 2D alternatives in the CK where applicable.

 

Also, the CK transitions between cells in downtown Boston so, so much faster now it's insane. Like, there's zero stutter now, where before it would take a couple of seconds.

 

E: Okay, I have the majority of the Financial district running at 30 or above FPS. There are still some problem areas, and I'm again beginning to realize that it's an issue with Bethesda's modeling.

 

LOD objects do not cull very well at all, I think everyone knows that by now. The problem is that many, many LOD buildings and parts have needlessly complex surfaces, the same way that a large number of high tech building panels are. Lots of instances of multiple triangles being used for flat surfaces.

 

The big issue here is that there are single groups of buildings that can produce around 8-9000 draw calls from just their LOD alone.

 

http://i.imgur.com/1LSKT9E.png

 

On the bright side, an entire regenerated world LOD is under 200 MB, and the architecture folders are under 20mb, so while it will be annoying to fix all of this stuff, it shouldn't be a huge file footprint, and will hopefully save some FPS around Boston.

Edited May 5, 2017 by Giggilyomeromicon

[mayhew2070]

@ zilav Thanks for the response. I suppose I understand a bit better now. That doesn't mean I like it though. What worked for Skyrim should have been re-used and improved upon. I mean, I don't see any improvements with the what they did for FO4 and I think I understand why mod authors might have more trouble working with it. Will take some time for me to learn more as most of this is very new to me. :smile:

IF Bethesda would've just explained all of this in the CK tutorial, that, that would've been just awesome.