Guide to non-loadorder related CTD and ILS

[darkmadmax]

For past 2 months I was spending much of my free time checking out the efforts produced by community since the last time I played skyrim (which was at release and in purely vanilla form), I must say even back then for me personally skyrim in vanilla state was a relatively stable game , though with occasional ctd here and there. Patched to most recent version vanilla was remarkably stable as well (I did not had a single ctd in one one 8+ hours play session)

But as I started playing with a lot of mods (my installed base is around 170 mods) they introduced significant instabilities/slowdowns , I did manage trough organized approach to root and pinpoint major main causes and solutions for most of them, so here Id like to share em and hopefully get some feedback too.

This article is not about the issues of intramod compatibility and load order related CTD here, but I focus on the systeml side involved in using mods and ways to troubleshoot, identify and solve CTD/ILS and performance issues . This is very much similar approach I always used as a system administrator and software developer for general troubleshooting , difference is just the scope of applications/problems.

We need 3 things : tools which allow us to measure performance/CTD affecting parameters, ways to stress test our system with most CTD prone scenarios , and approaches to solve the issues once they are identified.

The first step are means to monitor the game state (as far as it possible):

The tools:

--------

They should allow you to easily see the graph overtime of the several parameters which are good benchmarks for stability/performance:

- VRAM: gpu-z, sysinternals process explorer. Peak vram should be under 100MB from your max VRAM

GPUZ http://www.techpowerup.com/gpuz/

processexp from http://technet.microsoft.com/en-us/sysinternals/bb842062

- RAM: windows task manager/resource monitor, sysinternals process explorer . Peak RAM usage under 3GB (some folks say within 2.8GB)

- FPS: many tools, choose one with least footprint and most compatible one (e.g. NOT FRAPS). I use one from my videocard (MSI afterburner) , you can also use rivatuner (but it can not run at as enbboost dll). general benchmark is 30+ fps dips, with 60 fps at stable medium load (use fps limiter/vsync) . Under 15 FPS is a sign of stuttering(texture overload) and/or significantly overloaded rendering/shader engine. In some cases it could be also caused by severely malfunctioning papyrus scripts (check papyrus logs in c:\Users\<username>\Documents\My Games\Skyrim\Logs\)

turn on papyrus logging in system.ini

[Papyrus]

bEnableLogging=1

bEnableTrace=1

bLoadDebugInformation=1

We need reliable process of reproducing CTD/ILS/crashes :

Reproducing:

------------

Now as to reproducing -my main modmanager is NMM and I recommend it over everything else because of its high reliability, which does not involve virtual file system and symlinks (as is the case with "mod organizer") .It is also very good manager for texture/script mods as it can handle file level conflicts across multiple mods and cleanly uninstalls them when needed, something which is unreliable, cumbersome and time consuming to do by hand

Then you have to have "clean saves" - saves made in purest vanilla skyrim ,which never had any mods installed whatsoever, saved in safe location (indoors in an area typically not touched by mods) I do have one of my own (lvl 33 in riften orphanage), of which I also made another version with just vanilla DLC enabled. There is also several very good quality saves if you do not have your own or want more options: http://skyrim.nexusmods.com/mods/27357/? . I use very often the save in solstheim to test for TMS-syndrome (of which Solstheim ILS is a prime example -more on that in TMS guide). Needless to say you never overwrite those saves and use them to test mods load-out.

Establish baseline :

Have vanilla skyrim, set resolution and detail settings trough launcher. verify the parameters you want and run steps 2 to 5 of Stress Test . It is quite possible under heavy mod load you will have to reduce settings - particularly shadow detail ,AA ,anisotropy and texture detail so be conservative, or at least keep this in mind when doing heavy mod load testing.

On Tweaks:

skyrim.ini/skyrimprefs.ini tweaks:

There are very few hand made tweaks I personally use and most of them are game play tweaks (fov, arrowtilt) . I do not recommend any extensive tweaking of those files, keep em plain vanilla if stability is your goal. Use skyrim launcher to modify resolution/detail settings. its not that there are no working ini tweaks out there, its just that most of them are sacrificing stability without (IMHO) significant gains.

Core 3d party addons:

I use skse +one tweak plugin as "super stable" thing I almost never turn off . It is not that I am sure that they are 100% stable in all setups, but I did extensively test them already for stability and they never failed. Onetweak can possibly be dangerous in certain setups, but skse as a whole I think is very stable software on their own (that does not automatically mean that all skse using mods /plugins are stable - not at all).

Stress test:

-----------

The process is

1. install mods

2. load clean save

3. verify general baseline (vram/ram/fps) and stability . Do not proceed further until resolving those (check VRAM/RAM section for more details)

4. stress test - Flyover, spawntest.

5. ILS test (if ILS and /or flyover fails check TMS section of this guide)

6. (Optional) playtest - test various in-game mechanics such as npcs, items,scripts, perks etc (depending on the mods you installed)

If you find a breaking condition make a save as close to it as you can. use it as starting point for testing .

Then there are several simple scripts to help with stress testing :

night/weather cycle tests -set it and watch it change, it doesn't test weather that much but usefull day/night, NPCs going around - set timescale to 1000

Flyover. mainly to test outside areas ,fly in big circle from solutude to dawnstar, winterhold, windhelm, riften, Whiterun, Falskreak, rorikstead, dragonbridge, morthal:

 

It tests overall VRAM/RAM, landscape ,environment mods ,dynamic spawns ,etc . It can crash your heavily modded game much faster than running at normals speed on foot, sometimes you want decrease speedmult to 500-700. if it still crashes persistently without completing the flyover path in 1-2 tries I would investigate in details (is it same area? -what mods alter that area? are there dynamic patrols there? and so on). It is stress test and if you decide that you ok with its crashing the stress its your call but you are making stability trade-off. But personally if I cannot complete clean save flyover with dynamic spawns disabled I start taking out mods.

player.setav speedmult 1500
TCL

Spawn Stress - spawns mobs which fight player and each other.it serves well to test script heavy mods which touch enemies and objects (like bodies). In vanilla skyrim I can spawn 4 instances of that script without crashing (that is 204 mobs not counting summons). Under mod load its 2 to 3 instances . If I cant run this script at least once with combat resolving (takes them a while to kill each other , spoiler - vampires win )

//master vampire
player.placeatme 2eac1 10
//Giant: 
player.placeatme 936D3 10
//dragon
player.placeatme  F811A 2
//thalmor
player.placeatme f9614 24

ILS test - load save in solstheim and travel back to skyrim. Also travel to whiterun city ,save in outside cell(e.g inside the city ,but not in buildings). Exit game completely, restart and load that save.

General bethesda smoke test. Run it and open every chest, if there is any issues with items present in chest (and they contain most of armor/weapons), be it model/mesh etc it will crash on opening chest. this is more of a mod test , but it nice to do it once

coc qasmoke

If you passed stress test with flying colors... well you can stop right here ,otherwise there is more to the wonderful world of troubleshooting skyrim, , depending on what were results of your stress test:

VRAM

-----

In my personal opinion this is number one point which should be addressed when going for stable setup. Skyrim has a lot of graphical data, in fact it has more graphical data than fits in your vram. For it to properly work (especially on the consoles) it has dynamical load manger , which tries to load only what is needed at the moment. but what if "what is needed" is still more than your VRAM? -well then you have "stutter" when your fps is limited by load missing textures ->render frame->load more missing textures cycle. And because that manager/renderer chain is far from perfect it occasionally lead to straight up "fck th1s sh1t" - ill better CTD" scenario.

Installing a bunch of high res texture is a sure way to jack up your vram usage. In fact for me just using bethesda high res pack results in unacceptable levels of vram usage and stutter. But there is also a lot of choices and fine tuning with texture packs one can do to fine tune usage: there are different sets of textures -armor ,clutter, creatures ,landscapes ,dungeons and so forth. Installing all of them at 2k is likely ctd recipe for all but super beefy rigs (and even then you still will likely need to use enboost for RAM issue), but you can install just small set and/or one at 1k resolution or lower

What is good about VRAM is that it is simple to troubleshoot , monitor and resolve. - simply run VRAM monitoring program and make sure that its usage does not peak closer than 100mb off your max vram. Typically its best to measure peak vram usage outdoors, in big cities and spawning a bunch of mobs (=textures for their armor/bodies). for purely vram isolation testing I avoid whiterun though due to many other stability issues which are prone to happen in that area( but if you want - knock yourself out running 5 instances of spawntest!)

typical VRAM monitoring scenario:

http://static1.nexusmods.com/110/images/110/88369-1382673532.jpg

Aside of texture mods , there are other factors which influences VRAM - game resolution, anti-aliasing and anisotropy . Also chrome by default has hardware acceleration enabled and that uses 100-150 mb of your VRAM when its running ,and aero too (but I never use it) . I for example disabled AA and replaced it with FXAA as it saved me roughly 200 MB vram (and fxaa is faster on top of that), turning of chrome 3d acceleration freed me some more. All that enabled me to run several more 1k texture sets which improved the game looks . For dungeons though I run 2k set as there is much less overall texture load while indoors.

I must mention several good looking 1024 texture mods suitable for 1GB vram, many many thanks to their authors, they are:

Gamwich's Bethesda Performance Textures - http://skyrim.nexusmods.com/mods/43921/? and http://skyrim.nexusmods.com/mods/42360/?

skyrim HD lite version http://skyrim.nexusmods.com/mods/607/?

serious HD retexture 1024k ver http://skyrim.nexusmods.com/mods/2146/?

RAM

---

TesV.exe is a 32 bit process which is an entry point for all data loading operations (that includes all graphical data as well). 32 bit proceses are limited to 3.1GB max and from what I heard skyrim in particular becomes unstable at 2.8+ of used RAM.

In my personal setup this never happens, mainly because my VRAM is just 1Gb and it hit stability bottleneck first. However it is very real issue for those folks with beefier setups (2GB/3GB VRAM)

Therefore I must mention very interesting solution for this bottleneck if you ever hit it:

Vanilla skyrim loads all textures in its main memory before passing it to video card. That means if you running 2.5 GB of vram it will use 2.5GB of your system ram addressed by tesv.exe process just for graphics, which can easily push tesv.exe to its limits. There is a relatively well known d3d wrapper called enbboost, which among other things serves to circumvent this limit by offloading the managing of most texture data to another process (enbhost.exe). See this way your tesV.exe is largely free from having its addressable memory taken by textures .

Of course that implies that total amount of your system ram is more than sufficient for both tesV.exe ,enbhost and the rest of the system(that is at least 8GB)

To better illustrate that here is exact same save ,exact same scenario without and with enbboost:

http://static1.nexusmods.com/110/images/110/88369-1382673566.jpg

The ini file for that particular configuration(e.g. system memory offloading) of enbboost is located here http://skyrim.nexusmods.com/mods/38649/?. As of using enbboost for purely vram issues - I never noticed much of significant impact on it, when vram is overloaded it still crashes, enbboost or not, maybe slightly less and maybe with less stutters, but my benchmark for is NO vram related crashes, which I achieve without enbboost.

Before moving on the next issue, I must mention a generic approach to Bethesda games troubleshooting, and, well, in fact pretty much generic approach to systems troubleshooting:

DEFFS (Disable Everything For Fcks Sake!)

-----------------------------------------

and then quicksort....

This is when you throw in the towel and go to mod manager, select all non vanilla ESP/esm and disable them all. Provided the save was not in one the cells added by those esp/esm the game should load fine .(yeah disabling all 170 mods in one move). Depending on the mods u disabled sure the character/quests/whatnot could be in wonky state but game should load and run. I must stress though that save should not be using any of the assets of disabled mods, e.g. if it is saved inside new building added by one of the mods it will crash. -avoid making "safe" saves in places altered/and especially introduced by mods .

After that you start adding plugins back (of course while maintaining dependencies chains). 1/2 of disabled at a time. add ->load ->if it crashed disable 1/2 of enabled plugin and repeat process. when you find which half of plugins causes the crash on load you start dividing that half in half until you narrow it down to THAT MFCKER. -good old quicksort approach :smile:

TMS(Too Much Stuff) Syndrome:

-----------------------------

also might be known as whiterun ILS (infinite loading screen), Solstheim ILS,

This is when everything looks ok -your VRAM is peachy , your RAM is fine, you have good FPS and you know your script heavy mods are stable (or not even there at all). Yet you get infinite loading screen/freeze. And they typically the most prominent in those scenarios:

-starting game and loading a save in the exterior cell of whiterun.

-Travelling from Solstheim to skyrim

-more extreme cases - ILS when changing cell from interior to exterior especially in whiterun (e.g. leaving building)

What is more curious is that in less extreme cases of TMS if you load a save of an interior cell (or the less stressed exterior cell, typically outdoors) and after that load your " broken ILS save" it will magically load . And there is even ultimate insane "vodoo fix" which temporarily fixes even more severe cases(when leaving building causes ILS) -fix is opening the world map before leaving the cell(yeah its crazy but I have a save+mod load when this method is the only thing short of DEFF which allows to leave the cell - tested it over 2 dozen times, because I couldnt believe that "voodoo fix" actually works)

Of course the standard "DEFFS" fix works, but trying to find which plugin exactly caused it is to no avail. - very likely no matter which half you disable it doesnt cause ILS on its own. so what causes it? -simple answer is too many mods with high game data usage. Those are typically not your script/gamebalance and not texture replacers mods. These are mods which introduce NEW stuff - new sounds, new textures (as opposed to replaced one),new buildings ,new NPC+speech (like in 3d npcs) ,new level of details mods (for terrain ), NEW weapons and armor (like immersive weapons/armor). load one of them on its own - and its fine, but reach the critical mass of them and its starts happening

Personnally I started seeing ILS only after adding many content mods on top of already hefty setup of armor/weapons/encounter mods. And I want stress that its not that content mods are bad- I believr they are the best kind of mods there, because they increase replay value more so than any other mods. It is just they are the deciding factor in hitting TMS bottleneck, just like texture mods are deciding factor in hitting VRAM bottleneck

http://static1.nexusmods.com/110/images/110/88369-1382654036.jpg

see those mods with red marks? - enabling 1-2 of them together in any combination does not reproduce ILS in this particular mod load, yet enabling all of them does . I can decide which one of those I really want and/or go look for other things to disable(there are plenty of other candidates - distant details mods,sounds of skyrim,quest mods, immersive creatures/weapons/armors - all examples of high TMS impact mods)

What is a fix for it? -removing that critical mod mass is the obvious answer. "But but I want keep the all weapons/armor mods, Etac, 3dnpc, falskaar ,all LODs, those 4 quest mods, ultimate whiterun and and and and...." Well you might have to resort to use the above listed workarounds, you can also try to save only in indoors areas(especially in whiterun but avoid jorvaskaar). But do know you are making a tradeoff between stability and amount of things you want modded.

Why precisely skyrim chokes on that I dont know but if looking at root of vanilla data folder it takes ~5.5GB (bsa/esm/esp -thats without highrespack or DLCs). In my heavily modded variant its over 20 GB there, and then there are all additional subfolders there too. Skyrim simply is not very good at managing large amounts of data regardless whether it should "fit" (as both your RAM/Vram are not close to their limits). In fact I did extensive testing and no amount of vram usage/graphical tweaking could solve TMS when its happening. you can literaly drop your vram usage from 900mb to 200mb, your resolution too 800x500 from 1920x1200 and it will not fix infinite loading screen caused by TMS.

Perhaps some young bright fella , with lots of free time and a debugger in hand could dig to the root of the issue, but alas my own days with softice are long forgotten back in MS-DOS era. That "voodoo fix" though is peculiar thing - maybe TMS could be modded out of the game if the cause is found.

[rdunlap]

Well thought out! It fits my own experience too. I have been a developer since COBOL was state of the art in data and TMS *(eg too much sh...... um stuff) will bring any software/hardware/user/developer combination to its knees *(yes Virginia TMS is real). That said you should point out to those new to the mod scene that most of the people drifting around these sites are experienced modders and make the educated decision to mod anyway.

 

Some simple sanity rules:

 

1. till you are done....Done......DONE adding/changing/removing/tinkering/fiddling/ETC..... don't invest a hundred hours into a play through. you WILL be disappointed. besides how will you know the overall/true/total effect of the changes if not done one at a time and beginning from the same non skewed data-set? It gets said everywhere, but people have a bad habit of not seeing the large print. :| Never make changes mid game if you expect to continue effectively to a conclusion.

 

2. MOD AWAY!!! dont be afraid to mod. you WILL have CTD (Crash to Desktop), you WILL have ILS (infinite load screen), believe it or not you WILL have flying horses!! (outside my house lakeview estates, if you go from inside loaded save during sunny weather during daytime then visit island in lake swim across to the road and head home and the huntress comes trotting along at about 1500 feet above the road then zips off to the horizon only to come walking normally up the road from the original direction!!!!

 

2A. bonus perk points to anyone who can see from 2 what at least was the first best place to look for the issue i spent hours pulling out hair over! had even decided my big beefy NVidia had a bad peice of vRam. let that be a lesson to me. Remember to ALWAYS look at the simple first if only to tick it off the list of possibles. :) *(old fart debug axiom)

 

3. Look at/for the obvious but dont ignore the weird and wonderful. 2 was caused by physics limitations of the game engine along with a couple of mods that involved weather and sounds *(cool thunder(yes only happened during sunny weather)) when they were exposed to ENB with a particular preset. the physics in skyrim are tied strongly to the video stability

 

4. You probably haven't been the first to have a given problem, but you should be afraid of Google search on this type issue. it can be great for Grokking an issue fully but can also Grok you Crazy! there are hundreds or thousands of gamer out there who are dealing with the same issue and will gladly share their fix. Think about it. do you have a thousand evenings to go through each fix one at a time then play long enough to assure its fixed or not? I don't. Pick yourself a few .... FEW being the operative word here sites with good forums like this one and ahem steam to name a couple. Do some research first into the issue, I find people will give better advice and faster if they see I have spent my time wisely and can present the problem coherently.

 

 

The more you MOD the more often you ARE going to have CTD and or ILS so learn a habit of saving at the end of milestones you DON'T want to redo, and when your habit is well established consider turning off most if not all of the auto-save stuff since it can be flaky and if you learn to depend on it you will be sorry.

 

Robert

JAOF

[Antsh]

Probably worth mentioning Safety Load (http://skyrim.nexusmods.com/mods/46465)

 

Wasn't released when you wrote this guide, but it has helped my ILS issues immensely.

 

I was getting ILS b/c of SFO, even with the lower-res LOD texture and that mod fixed it.

[rdunlap]

New one for me. :)

going to look at it and see if they are handling memory stacks directly or just handles....

thanks and your right needs mentioning with a small warning. use this once you use dark_madmax procedure first to eliminate as many issues as possible then drop this in place and enjoy. :)

always better to squash bugs than to throw a cover over them.

 

Robert

[mindw0rk2]

I actually read everything and the last point what I came to after more then a hundred hours of sweat and blood, trying to make my heavily modded Skyrim work. I did everything. Cleaning, BOSSing, ini tweaking, patching, etc. Eventually it became clear - game just cant handle it when there are too many big mods. Proof?

I have a pretty stable build at 180 or so mods. No crashes, no freezes. If I try to add more mods over this amount, suddenly all kinds of problems appear. Those mods I add arent even script heavy, they totally different and cant cause issues by themselves. Still they do if I install them on heavy compilation.

 

Its a huge problem and I hope somebody will solve this. If some genius does, we could make our dream Skyrim. For now - its only choosing which mods you need the most

[rdunlap]

you *(meaning we) are running into an inherent windows issue too sir. windows gets funky when an app gets greedy with file/memory handles. Its been an issue for many years and many windows versions/updates/fixes that weren't. Basically not going to fix either, to many fences been in play too long. Its like the "Old Hands" at modding say, decide on what you cant live without and stick to it. any deeper modification must be done at the root level *(game developers themselves) its great that they give us this to keep us busy between ES releases right ?

 

Robert

[rdunlap]

Antsh,

Safety load is doing as advertised. seems to play nice with everything else too. just got to run it a while and watch. I loaded it as a plugin to SKSE and went to a save area that wont work for me 9 of 10 times if i reload it and Voila' no CTD :)

 

Robert

[rdunlap]

Any other old farts like me out there that remember Quest for Glory series? originally "so you want to be a hero" i think. It was Great for its time but more unstable than anything you could run today but that still didn't stop me from playing thru all 5 games *(still fire up emulator and play at times)

 

Robert

[rdunlap]

Dark_MadMax,

That map thingy is strange stuff. Its real though. I have a save in Lakeview that for some reason in my rig/mod combo is very unstable and I can load it and then open the map for a moment and this silly thing will act like nothing is wrong. :) You got to love the weird and wonderful to fully enjoy gaming. maybe it causes a Ram or vRam flush call?

 

Robert