PSA: IMPORTANT! Read this before deleting LocalShaderCache!

[Amineri]

The current SDK build has a small flaw that can result in mod dependencies if you delete your LocalShaderCache file.

 

This shows up only if you are building more than one mod (like... we have 3 mods at LWS). It can happen even if only one mod has 3D assets.

 

SHORT VERSION:

 

• As a workaround
  • Delete the "cached" mods from the \Steam\SteamApps\common\XCOM 2 SDK\XComGame\Mods folder before doing each mod build.
  • Delete the file \Steam\SteamApps\common\XCOM 2 SDK\XComGame\Content\LocalShaderCache-PC-D3D-SM4.upk before doing each mod build
• Leave the mods in your game folder \Steam\SteamApps\common\XCOM 2\XComGame\Mods alone.
• Leave the mods in your ModBuddy project folder alone.

 

----------------------

 

LONG VERSION:

 

I'll write up more here about what's going on under the hood later, when I have some more time ^_^

 

Here's what's going on in a bit more detail...

 

The precompileshaders commandlet works in the following manner :

1. Launch the SDK executable
2. Load up existing shaders (e.g. RefShaderCache, GlobalShaderCache)
3. Load mod asset packages, one at a time
4. When loading mod asset packages, if any fail to find needed shader in already loaded shader caches, then the shader will be "compiled at run time" with the results added to the LocalShaderCache -- this is standard Unreal behavior (https://forums.epicgames.com/threads/695400-Missing-cached-shader-severely-slowing-down-maploadtime)
5. After all mod asset packages have been loaded, the LocalShaderCache is copied and renamed to be the ModShaderCache in the mod folder
6. When game launches, any mods that have been specified as active through the Launcher/ModOptions.ini file have their ModShaderCaches loaded and added to the common shader cache pool in memory

----------------------------------------------

 

Why to delete the LocalShaderCache

 

If you don't delete the LocalShaderCache, then the previous LocalShaderCache is merely added to and then copied to become the ModShaderCache. This results in file bloat, particular if LocalShaderCache happens to be quite large. The release LocalShaderCache was about the same size as the release RefShaderCache, (~85 MB), and this will be added to any mod, regardless if it needs any shaders or not. If a mod has content assets, but doesn't generate any new shaders, I've found that the resulting "empty" ModShaderCache will be 371 bytes.

 

This didn't happen with the LWS release-day mods because the pre-release build of the SDK we were working with didn't have the really large LocalShaderCache. Those mods ended up with a ModShaderCache of ~233 KB, which is actually still a bit larger than it needs to be, but I'll elaborate on that late.

 

One downside to deleting the LocalShaderCache is that it can also be used by UnrealEd. UnrealEd is actually a upk, running under the game engine. So it uses shader caches just like the game does. If you are working with new materials and UnrealEd can't find a shader, it will compile the shader at run-time (with the result being added to the LocalShaderCache). Compiling shaders at run-time makes UnrealEd load slower.

 

---------------------------------------------

 

Why to delete "cached" mods from the SDK Mods folder

In step 2, when the precompileshaders commandlet is loading existing shaders, it appears to load any ModShaderCaches it finds in the SDK Mods folder. Evidence for this theory is as follows :

• When launching UnrealEd with cached mods in the SDK Mods folder, it can be observed that during the loading splash screen that the ModShaderCaches are being loaded
• I conducted the following experiment :
  1. Start with empty SDK Mods folder
  2. Build mod that requires shaders -- I used LW_SMGPack
  3. Observe that the LocalShaderCache file is created, as well as ModShaderCache -- this file was ~133 KB
  4. Do not delete the LocalShaderCache
  5. Build mod that does not require shaders -- I used LW_OfficerPack
  6. Observe that the LocalShaderCache file stays the same size, and the OfficerPack ModShaderCache is also ~133 KB
  7. Delete the SDK Mods folder LW_SMGPack
  8. Delete the LocalShaderCache
  9. Do not delete the SDK Mods folder LW_OfficerPack, and leave its ModShaderCache
  10. Rebuilt the LW_SMGPack mod
  11. Observe that the LocalShaderCache is 371 bytes, and that the LW_SMGPack's ModShaderCache is also 371 bytes
  12. Launch game with only LW_SMGPack mod enabled, in debug mode
  13. Observe that redscreen messages about missing/incomplete textures appear, and that some texture are "corrupted black"
  14. Close game, relaunch game with both the LW_SMGPack and LW_OfficerPack enabled, in debug mode
  15. Observe that there are no such redscreen messages, and that the textures display correctly

According to the theory, what is happening is that in step 5, the shaders for the SMGPack were already present in the LocalShaderCache, and so get built into the ModShaderCache for the OfficerPack. Then, when I delete the LocalShaderCache, and rebuilt the SMGPack in step 10, the precompileshaders commandlet loads those shaders from the OfficerPack, so it doesn't add any shaders to the LocalShaderCache, with the result being that the SMGPack's ModShaderCache is empty, and doesn't contain the shaders needed by the mod.

 

Thus when the game is run and SMGPack runs by itself, I get the following redscreens/log messages :

[0021.84] Log: Missing cached shader map for material Laser, compiling.
[0021.84] Warning: Redscreen: Compiling Laser at run-time, please ensure this is part of the cooking process
[0023.13] Log: Incomplete cached shader map for material WeaponCustomizable_TC, compiling.
[0023.13] Warning: Redscreen: Compiling WeaponCustomizable_TC at run-time, please ensure this is part of the cooking process

The WeaponCustomizable_TC is the base material used for weapons and weapon attachments. I also created a duplicate of the Laser material in the package to support the Laser sight attachment. Visually, the effect is that of blackened, darkened textures.

 

When the two mods are both run, the needed shaders for the SMGPack are loaded with the OfficerPack's ModShaderCache, with the result that the textures display as expected. However, the process has created an undesired dependency between the two mods.

 

By deleting the cached Mods from the SDK Mods folder, it ensures that the precompileshaders commandlet won't load them when it runs, and that the ModShaderCache built for a particular mod will include all shaders that the mod actually needs.

 

In separate tests with the Muton Centurion, I would instead get the following redscreens/log messages :

[0058.84] Log: Missing cached shader map for material Alien_SD_SSS, compiling.
[0058.84] Warning: Warning, Failed to compile Material LWMutonM2.Materials.Alien_SD_SSS for platform PC-D3D-SM4, Default Material will be used in game.
[0058.84] Log: Missing cached shader map for material Alien_SD_SSS, compiling.

The visual effect of "Default Material will be used" is that the Unreal default blue/white checkerboard material will be used instead of the correct one.

 

------------------------------

 

The LWS release day mods were released before we knew about having to delete the LocalShaderCache. However, the SDK at that point didn't have the really large existing LocalShaderCache in it.

 

Thus, as we would build each mod, all of the shaders required by any of the mods (only Centurion and SMGs needed any shaders) would be accumulated into the LocalShaderCache. That LocalShaderCache would then be copied to become the ModShaderCache for each of the 3 mods. The end result is that each of the 3 mods contains all of the shaders necessary for the entire collection of mods.

[Yzaxtol]

Yup, this hit me today after I re-wrote my code and I was like:

 

WHYYYYYYYY! My mod has been destroyed.... but suddenly:

 

Aminery uses Magic .... it's super effective!

 

 

Yay

[CaveRat]

Thanks. Very useful.

[Lucubration]

This is very good to know. Thank you.

[Amineri]

I've updated the "detailed" section of the original post to provide more insight about what's going on.

[Zyxpsilon]

Only one major question still remains unanswered... Can Firaxis Devs find a proper ModBuddy/SDK solution (patch) that would solve all those problematic issues in one big swooping official correction?

 

It's one thing being able to recognize what and how to repair everything ourselves (( with solid facts provided by our dear Amineri! ))... another to just trust a Modding tool has been reliably setup by its producers for good.

 

:ninja:

Edited March 6, 2016 by Zyxpsilon

[FxsRMcFall]

XCOM 2 Modbuddy / SDK will be getting updates / fixes alongside the game. At the very minimum the SDK must receive an update to keep the script code in sync with any patches that come out, but we will roll as many bug fixes and QoL improvements as we can into the updates as well. Issues such as the mod shader cache stuff mentioned here are an example of the type of issue that would be addressed asap.

[Amineri]

Thought I would bump this in light of today's patch.

 

The patch to the SDK does appear to include the auto-deletion of the LocalShaderCache from SDK Content folder. So it's an improvement.

 

However, the precompileshaders commandlet is still loading ModShaderCaches from other the SDK Mods folder. So if you happen to have two mods that both build the same customization of a material instance, there can be cross coupling between the shader caches of the two mods. Hence you still need to at the least delete the ModShaderCaches from other mods when building.

 

This is a bit of a corner case, but is something I was able to reproduce in a non-pathological use-case.