Frontier Size Issue / FNV Maximum Records Documentation

[kylemarshiku]

Hello. We have a huge game install that some of you might be aware we documented in this guide. For those of you not aware, we have ideological deviations from some of our critics in the way we mod our game, but this is not about that. Our game is massive, with a prioritization on taking any mod we want and smashing them all together. We are very satisfied with the result, and we fix errors as we stumble across them. Our game has large mods like Fallout New California (FNC) and Tale of Two Wastelands (TTW) installed, and we made a myriad of patches for anything else that was necessary. Prior to the attempt to also incorporate The Frontier (TF), everything worked fine. And that brings us to what we'd like to talk about and post here today.

 

boring specs and stuff

 

CPU: Intel i7-8700
CPU Speed: 3.20GHz
Graphics Card: NVIDIA GeForce GTX 1070 Ti
VRAM: 8192MB
Resolution: 1920x1080
Ram: 16GB
HDD Free Space: 367GB (just for game drive)
Laptop: No
OS: Windows 10 21H1 64-bit
Platform: Steam
Location: D:\Kira\Program Files\steamapps\[etc]
Game Version: Latest
DLCs: All
Mod Manager: Mod Organizer 2 2.2.2.1
ENB: Yes
LAA: Yes
FPS: Default 30
Mods: See guide

 

 

Hypothesis: There are a maximum number of game records that a given save file can handle, and upon experiencing an overload, amounts to a sort of out-of-memory crash.

 

Perhaps we are wrong, but we have done extensive testing and are not sure what else to make of this. Perhaps someone reading who knows more than we do can enlighten us. The problem is that we have gone through great effort to bypass and ameliorate every other limiting factor there is to make an extremely massive Fallout New Vegas, and so we feel like we've hit upon some type of hidden limit that few others experience. Again, perhaps an expert here knows about this limitation, and it's just a limitation most don't talk about because most people never make it this far. We admit that that is totally possible.

 

Issue: A heavily-modded game that works fine has The Frontier mod added to it. Upon its addition, it mostly works perfectly fine, but the game crashes whenever brought to the larger settlements in the Mojave. Primm, Novac, and Freeside all CTD.

 

Please note that the below tries aren't literal one-time off tries. Each try below indicates NUMEROUS launches and tests of varying orders. We're just trying to be categorical in our explanation.

 

Try 1: So thus began our testing to resolve this issue. We had previously not met an issue we couldn't match. Can't be done? Bah! If make a patch we must, then we shall! We started by making sure that FNC, TTW, and TF could all be installed simultaneously. We made a clean profile via Mod Organizer 2 and only installed those mods, along with the patch to make FNC and TTW work together. And... it worked perfectly fine. Since TF is compatible with both FNC and TTW independently, this was not surprising. So why did our game not work? Well, you're probably thinking another mod we had was conflicting in some way or causing issues. We suspected that as well.

 

Try 2: We began trying to identify conflicts. We tested launching the game and going to the crash sites repeatedly, turning plugins on and off, brute-force trial and error. However, to our surprise, we discovered that the issue was with... NONE of them. We tried turning on and off every plugin, even asset mods, but nothing. Well, we did discover a curious thing. Our game currently has almost 120 plugins, just under the legacy plugin limit for FNV. If we turned off several of our larger plugins, it worked. The crash sites would not induce crashes when TF was installed. The problem is that it did not matter WHICH plugins we turned off. We have the mod that supposedly lifts plugin limits (even though TF also includes that fix), but still we suspected we must be experiencing some type of plugin limit.

 

Try 3: We tried to reduce our number of plugins without sacrificing all of our content. We turned off a dozen plugins, far more than we had done earlier, but these plugins were much smaller in scope. Things like the Take Your Time mod and other small self-referential script mods that we were unable to merge into bigger plugins. Despite turning off many more plugins, the crash sites remained active. Reducing plugins was not the answer, not unless they were sufficiently "large" plugins, like some of our largest merged plugins.

 

Observation: By this point, we're starting to notice that there's some type of limit we're coming up against. Note that we can go into TF's world and start doing quest stuff. It's just that having it installed makes some of the more well-known and presumably heavily-modded areas in the Mojave get overwhelmed and crash. This is strange considering that TF doesn't modify these areas at all. We don't know what wall we are pushing against, as it isn't plugins or conflicts, but we start calling it Stuff. The game doesn't seem to like how much Stuff we have.

 

Try 4: We thought that Stuff might be other assets in the form of meshes, animations, and textures. So we turned off our biggest asset mods to see if that made a difference. It did not. Note that we are familiar with what an Out of Memory error looks like, the kind of error that usually accompanies using too heavy textures. It's why we don't use 4K textures. At no point here do we get an Out of Memory error. Just CTDs.

 

Try 5: Having exhausted ideas on the plugin side of the equation, we turned to the save. Perhaps something is wrong with it? We play very methodically and almost never have removed a mod from our current save. We're over halfway done with TTW and have already completed FNC, and so our save is around 40 MB. From searching around, that's not an unusual size for a save late-game in vanilla, but certainly bigger than average. We start testing different saves, and we discover quickly that making a brand new save and heading straight to a crash site... works. It doesn't crash. We would've tested this earlier, but you understand we're invested in making our current save work. So now that we've determined a fresh save works, how did ours go wrong?

 

Try 6: We began testing various saves we have, because we have kept an archive of every save we've done. Our most recent save is just over number 2000, and that one doesn't work. So we try earlier ones and discover that we had to go waaaaaay back about a thousand saves to find one that works and does not crash in a crash site. We searched to try to find a single moment where we might've F'ed up in the saves, maybe find some odd incompatibility, but we couldn't find it. The strangest thing was the gradual nature of it. In between save 1000 which definitely did work and later saves that definitely didn't work, were a bunch of saves that kinda-sorta worked. They'd only crash in the crash sites sometimes or would load for a minute and THEN crash. Usually it simply crashed during loading. This gave us the impression that there wasn't a single moment of incompatibility that tainted the save. Rather, it seemed like at some point, our save have experienced too much Stuff.

 

Note: Despite our failure to identify an issue with our save, we wonder if playing through the game with TF installed from the very first, initial save would solve the problem. Perhaps, though adding mods part-way through is usually not a problem. It's removing them. For some mods, adding part-way might be an issue depending on how its implemented, but TF is kinda plug-and-play, altering very little outside of its added worldspaces. We'd like to test this, but we can't. The only way to really test this would be to start a new save and play through 1000+ saves with the mod on to see if we could replicate the issue. We've played this game over a year, and testing this is sadly not feasible. We're forced to presume our save is fine, and considering the lack of a single moment of incompatibility, we wager that is fine. We believe trying to play through it all again would lead to the same result with it not working given a long enough period of time.

 

Try 7: We tried cleaning our save using New Vegas Save Cleaner to see if it could junk any data or make it smaller. It had no effect whatsoever. We went through the whole process, followed the instructions. Just nothing. We presume that perhaps our save is already cleaned via some other mods and simply has little junk to remove. Either that, or the cleaner is simply outdated and of no use to us.

 

Try 8: We thought that perhaps, since removing large mods makes it work, that we could do that and only then put them back in. But no, putting them back into the save makes it not work again. Crashes at the crash sites. No matter where TF is in our mod load order nor plugin load order. Removing and adding back in makes no difference. So now we were really stumped.

 

Try 9: Having watched the Task Manager considerably on the other screen as the game loaded, we began to consider the numbers might bear some significance. Our game is Large Address Aware and utilizes the 4 GB feature in the executable, as is standard for TTW and TF. We noticed that every time the game crashed, the memory used never reached nor exceed 4 GB. It'd get higher than 3.8 and then crash quite frequently. We thought maybe that was the Stuff, a memory limit. Perhaps we were just achieving an Out of Memory error in a very different manner than usual. Thus we began tests to try to lower that number, which largely revolved around turning specific big mods off. Our results weren't very helpful, as the only thing that worked was turning off the big merges as already addressed. After further testing, we deliberately shot up a heavily populated area elsewhere in the game, cranking the memory up, and reloading the game. The game loaded successfully exceeding 4 GB of memory, according the Task Manager. Having made no progress, we decided this might have been a ghost we were chasing and to change tactics.

 

Try 10: Having looked through our game extensively and coming up empty, we decide to look into TF itself. Perhaps the issue isn't us! Perhaps there's just SOMETHING in TF that is overloading our game. Maybe some kind of soft incompatibility. We start by removing what seem to be the most likely culprits: Quests and Scripts. It doesn't work. The game loads, but the crash sites still crash. We tried removing EVERYTHING in the plugin, making it a dummy plugin. Naturally, it works now. So we theorize that maybe deleting the right thing here could be the answer! Maybe we can trial and error this until we narrow down the problematic records in TF.

 

Try 11: And so we tried that. Gutting a category of records, sometimes multiple categories. Trying, failing, restarting, replacing. We had a backup file we'd use to bring records back after each test. We wrote down each combination we tried. We tried deleting half the records to see which half it was in, and we ran into issues. Doing this didn't always result in a binary it worked or it didn't work. Sometimes the game wouldn't load. We're dissecting TF; it's not unexpected. But after elaborate testing, we managed to narrow it down to Cells, Creatures, Dialogue Topics, NPCs, and Worldspaces. Deleting those would make it work. Leaving them in would make it crash at the crash sites. So we know we need to look in one of those. Also, side-note, we had to delete Form ID Lists each time or the game wouldn't even load.

 

Try 12: We discovered that it didn't matter which combination we did. It we restored any one of those categories, the game would work. But if we restored 2 or more, the game would not work. This was the worst news we could have received. It meant that there wasn't simply an incompatibility anywhere in TF at all. This is effectively a microcosm of the plugin tests from earlier. The only reason narrowing it down to these categories had worked was, we deduced, because they were the largest collections of records! In short, it seemed that the only thing that worked was a reduction in Stuff. It mattered not which stuff that was.

 

Try 13: Starting to get desperate, we look for any other possible answer. We noticed that for some strange reason, nearly every single quest in TF is Start Game Enabled. We compared it to the DLCs, which in turn had nearly NO quests Start Game Enabled. We thought, hey, maybe this is a huge drain on the game! We made a patch that turned off that flag for the majority of quests (with the intention of, it this worked, making our own script and trigger that would turn them on when necessary). It didn't matter. Still didn't work. Crashes still.

 

Try 14: We know there is some wall fighting against us, just not what it is. This Stuff. We begin to feel that the Stuff might simply be the amount of content, i.e. the number of records loaded. We look in FNVEdit at the records per plugin. The highest four plugins in order are: The Frontier.esm (700,000), Fallout3.esm (600,000), FalloutNewVegas.esm (500,000), and Merged Plugins Minor Quests.esm (400,000). We decided to make a duplicate of that merged that we called Merged Plugins Minor Quests 2. There's a high risk of something going wrong, but since it is a merge that just adds quest mods, we rationalize that it should just add doubles of every quest, NPC, and everything. The game should, hopefully, load. but alas it did not, which was inconclusive. If we could download another 700,000 worth of records to test, we could test whether that mod worked. If it didn't, it would verify that these records were indeed the Stuff that held us back. If it did work, then it would imply something uniquely wrong with TF. But, we ran into a problem... There is no mod to test. No mod compares to TF's size. None. Not TTW. Not FNC. Even our largest merge was barely more than half its size. And we don't even know what we could merge to rival that.

 

Try 15: With nothing left to try, we do some desperate Google searches. We've already scoured the GECK wiki. We end up on TF's Nexus page. We see the optional mod for weather performance. We try it. No effect.

 

Try 16: We make this post.

 

Conclusion: It would seem there's a maximum amount of Stuff that can be in a game, and that figure is somehow correlated with the number of records currently in play in a given save. It would seem that a fresh save is capable of doing anything, but not EVERYTHING in a single playthrough with the upper threshold of records reached. It's conceivable that the game just starts breaking down at that point. If we kept TF installed in our game, it is possible that continued playing on this save would result in even lesser settlements in the Mojave to crash, increasing the number of crash sites. So, in truth, the hypothetical maximum of records would be the number at which it is only a matter of play time until major settlements beget CTDs. Going under that number will ensure that all areas are accessible, given no other issues, and going over that number will increase the rate of instability and accrual of crash sites, starting from the most complex areas.

 

Future Plans: Ideally, we're hoping someone smarter than us can shed some light on our issue. We're hoping our detailed explanation of tests will allow people to treat us in good faith and assuage any fears that we're seeking trolls just because we deviate in our play style. Hopefully the responses will be free of trolls and instead have genuine concern, but don't worry, we're not expecting anyone to fix our problem. We knew what we were getting into, and we've been happy with it. We just really wanted the Frontier to be part of that experience, and it seems like we've hit some kind of limit. If anyone knows, please reach out. Otherwise, perhaps we have discovered something that might be useful to know. We could calculate our total number of records if anyone is curious to know. Sadly, if we can't resolve this, we'll just have to resign ourselves to not partaking in the Frontier or sacrificing nearly a hundred other mods to make room for it,... and that just stinks.

 

Thank you for your time.

Sincerely,

Kyle and Kira

Edited September 25, 2021 by MysteryMachineX

[M48A5]

TLDR; Yes, there is a limit to the amount of records that the game engine can handle.

 

The theoretical limit is 255. However, this is impossible to reach without using the Mod Limit Fix Plugin.

 

This will enable the game to handle more mods. The exact number depends on a lot a variables and the total number of files cannot be positively stated.

[Pixelhate]

@ M48A5 I think you're are confusing number of records and numbers of plugins

[kylemarshiku]

ExhalePit is correct. M48A5, you're confusing records and plugins. Records are the amount of entries in a given plugin. And you're describing the plugin limit. While other Fallout games have a plugin limit of 255, FNV has a floating plugin limit that tends to hover 120-130. On Try 3 above, you can see we tried to assess whether we needed to lower our plugin limit to a considerably lower figure, but even after turning off a dozen plugins, there was no change. Also keep in mind that we said that we were using the mod that supposedly bypasses the floating plugin limit, meaning this should already be addressed anyway.

 

That being said, we posited that there may still be a limit of records even when bypassing the limit of plugins, and so you might be indirectly correct anyway. We are not certain.

[Pixelhate]

Asking knowledgable folks, I got this possible & theoretical answer: FEFFFFFF
That would be 4,278,190,079.

[kylemarshiku]

Hm, that doesn't add up. 4 billion records? We're not coming close to that number. We added them all up. Not counting the Frontier, we have 5,555,091 records. When the Frontier is added, we have 6,315,796. So, 6 million is well under 4 billion. Perhaps it counts assets and BSAs as well? We could count that up, too? Would that count loose files as well?

[Pixelhate]

If I understand this correctly, FEFFFFFF is a mathematical hard limit, you can't go beyond that. but it is very possible that the engine puts a limit (way) before that, which is so far unknown.

[kylemarshiku]

We had an idea, and we just tested something else. So we'll add this:

 

Try 17:

 

Before we can explain what we tried, we have to explain why we tried it. When making our large merges with the Merge Plugins program, we discovered that the merges simply wouldn't work as ESMs without a sort of ESP "anchor". This has to do with what records ought to be in an ESM versus an ESP. Surely some mod authors are familiar with this recommendation, as we have seen mods that come with two plugins, one ESM and one ESP. This happened with FOOK and Adult Club New Vegas, off the top of our heads. Since we aren't making these plugins from scratch but rather are just merging existing mods, our solution to this problem was to use FNVEdit to Deep Copy Override a few categories (Activator, Cell, Talking Activator, and Worldspace) into an ESP per merge. These are the anchors, and they work! Doing this procedure allows the merges to function. Removing them, they cease to function. We only did this for merges that HAD to be ESMs (like mods with non-white NPCs, for example), like Merged Plugins Major Quests and Merged Plugins Companions, but NOT Merged Plugins Perks nor Merged Plugins Apparel.

 

So we wondered... What if we treat TF like a merge? It seems to have been constructed that way. So we made an anchor for TF, doing a Deep Copy Override on those four categories of records as we have done for past merges. What this means is that we effectively just added a plugin to our game that makes ZERO changes to the game. It should NOT break ANYTHING. But you know what it does do? It adds about another 500,000 records. And guess what? The game crashes upon loading in ANY location. The entire game is a crash site now!

 

This seems to confirm our suspicion that Stuff is DEFINITELY correlated to the number of records in the game. It may simply be that 6 million is the limit, but again, it's not a hard limit, as it only starts with some areas becoming crash sites. It is entirely possible other things factor into this, like assets or BSAs, which would be interesting to know.

 

At this point, three pieces of information could be helpful to us. 1. Does anyone think BSAs or loose files could help this situation? If it's counting BSAs, then fine, let's unpack them! If it's counting loose files, then fine, let's pack them into BSAs! 2. Does anyone have any better idea of what this Stuff might be? Narrowing down the problem could be useful. 3. Does anyone know why this anchoring effect is specifically helpful? Perhaps there's another way we could go about it that would cut down on the number of records in our game.

 

EDIT: One idea we have is this. What if we could change the Cells and Worldspaces to be actually IN the anchors? If they're referenced by something, we can't do that, as it'd be a circular reference. But maybe for the rest of them? The problem is that that would be an extremely tedious process. Perhaps an FNVEdit script could be written that would check each Cell and Worldspace sub-block and move them into a separate plugin, but we know not how to do that kind of scripting. But, this wold effectively accomplish the anchoring effect with over a million fewer records.

Edited September 25, 2021 by MysteryMachineX

[JimboUK]

It was a long time ago so my memory on this is more that a little hazy but wasn't there a bug in Oblivion that caused FormID's or RefID's not to be recycled so you ran out of them? although if I remember correctly all it did was make inventory items vanish. New Vegas is on the same engine albeit a updated version, maybe there's still an issue?

 

I do know that New Vegas does fall apart slowly as you add more hours to your saves, one fix I've found for that is to go somewhere where there are no NPC's, Jean Sky Diving for example, wait four in game days, make a save, exit the game to make sure that nothing is lurking in memory and then restart from that new save. I assume what it does is reset a lot of stuff, which should free up ID's.

[kylemarshiku]

Is this the bug you're referring to?

 

https://en.uesp.net/wiki/Shivering:Reference_Bug

 

Unfortunately, we're not sure how that information can be of use. Implementing anything akin to a formal patch provided by Bethesda is surely outside our means, isn't it? But perhaps that could be a possible problem. Perhaps the Stuff is the game, for some reason, neglecting to properly recycle the IDs because of how many IDs are being used.

 

We did try to do your suggestion of going to a cell with no NPCs and waiting 4 days. (We had to go to a different cell, because someone is in Jean's Sky Diving in our game.) But unfortunately, waiting, saving, exiting, and reloading didn't have any effect. The crash sites still crashed our game after doing all that.

 

Currently we feel the best step forward would be to figure out how to make an FNVEdit script to move REFs into an ESP. We've been trying to figure out if that script already exists, but we've yet to find it. OTherwise, we'll need to make it (or more realistically, find someone to make it).

Edited September 26, 2021 by MysteryMachineX