R&D - Materials

[Drakous79]

It may look crazy to start this thread so close to XCOM 2 release date, but I feel it is something we should try to solve.

 

Missing or incomplete cached shader maps may be the only obstacle keeping us from new models without animations. If you ever tinkered with mats, you could see similar compile error in game's Launch.log:

Log: Missing cached shader map for material CHH_Soldiers, compiling.
Log: Can't compile CHH_Soldiers with seekfree loading path on console, will attempt to use default material instead
Warning: Warning, Failed to compile Material Instance CHH_Lv1MedFem_MOD.Materials.MInst_Lv1MedFem with Base CHH_Soldiers for platform PC-D3D-SM3, Default Material will be used in game.

Basically the log is saying, that CHH_Soldiers material defined in CHH_Lv1MedFem_MOD.Materials.MInst_Lv1MedFem does not have its shader map in RefShaderCache-PC-D3D-SM3.upk.

 

 

What is shader map? What the cache?

 

Hopeffuly following quotes can explain. I am not certain how relevant they are, because this all is new to me.

 

Source: https://docs.unrealengine.com/latest/INT/Programming/Rendering/ShaderDevelopment/index.html

Global shaders
Global shaders are shaders which operate on fixed geometry (like a full screen quad) and do not need to interface with materials. Examples would be shadow filtering, or post processing. Only one shader of any given global shader type exists in memory.

Material and Mesh types
Materials are defined by a set of states that control how the material is rendered (blend mode, two sided, etc) and a set of material inputs that control how the material interacts with the various rendering passes (BaseColor, Roughness, Normal, etc).

Material Shaders
Shaders using FMaterialShaderType are pass specific shaders which need access to some of the material's attributes, and therefore must be compiled for each material, but do not need to access any mesh attributes. The light function pass shaders are an example of FMaterialShaderType's.

 

Shaders using FMeshMaterialShaderType are pass specific shaders which depend on the material's attributes AND the mesh type, and therefore must be compiled for each material/vertex factory combination. For example, TBasePassVS / TBasePassPS need to evaluate all of the material inputs in a forward rendering pass.

A material's set of required shaders is contained in a FMaterialShaderMap. It looks like this:

FMaterialShaderMap
    FLightFunctionPixelShader - FMaterialShaderType
    FLocalVertexFactory - FVertexFactoryType
        TDepthOnlyPS - FMeshMaterialShaderType
        TDepthOnlyVS - FMeshMaterialShaderType
        TBasePassPS - FMeshMaterialShaderType
        TBasePassVS - FMeshMaterialShaderType
        Etc
    FGPUSkinVertexFactory - FVertexFactoryType
        Etc

Shader caching and cooking

Once shaders are compiled, they are stored in the Derived Data Cache. They contain, in their key, a hash of all the inputs to the compile, including shader source files.

That means that changes to shader source files are automatically picked up every time you re-launch the engine or do a 'recompileshaders changed'.

When cooking assets, material shaders are inlined into the material's package, and global shaders are stored seperately in a global shader file which allows them to be loaded early in the engine startup.

Source: http://www.neoaxis.com/wiki/Documentation/Articles/Overview_of_Shader_Cache_Compiler

Each matrial has its own shader generation settings. Aside from settings of the material, the code of the shader depends on global settings of the engine. For instance, on the selected type of shadows. When the engine generates material, it queries the cache and checks for the given shader taking into account parameters of the material and global settings. If the shader is found in the cache, the engine uses it, otherwise the shader needs to be compiled in run-time of the application.

To avoid compiling shaders in run-time, a cache is created. It contains generated shaders for all possible combinations of global settings of the engine.

 

 

XCOM's RefShaderCache-PC-D3D-SM3.upk

 

It is compressed upk containing Name, Import and Export table and loads of serialized data.

 

NameTable contains names of material parameters. Maybe you know CMOD and CMODB referred by scripts changing armor tint.

 

ImportTable has two entries:
0xFFFFFFFF (-1) ( FF FF FF FF ): Class'Engine.ShaderCache'
0xFFFFFFFE (-2) ( FE FF FF FF ): Package'Engine'

ExportTable has one entry:
0x00000001 (1) ( 01 00 00 00 ): ShaderCache'CacheObject'

 

Serialized data consist of concatenated chunks of shaders and material shader maps.

 

EU and EW RefShaderCache-PC-D3D-SM3.upk header differs:

• EU LicenseeVersion: 59 / NameCount: 1563
• EW LicenseeVersion: 64 / NameCount: 1796

 

Material and material instance object format

 

There are some default properties and parameters. Interesting part is at the end, after <None> tag.

 

I compared EU / EW Soldier_FemaleKevlar_SF.upk, because armor material (CHH_Soldiers) fails, but country flags (CHH_Flags) are working.

 

EU and EW CHH_Flags parent material contains the same GUID. The last 4 bytes seem to be the same for all or most materials in given upk file, but are different in other upk files, where is CHH_Flags parent material defined.

 

CHH_Flags parent material:

// EU CHH_Flags	
<None>
01 00 00 00 // 1 entry follows
31 97 D4 1F A8 78 73 46 96 4C 32 1C 64 C7 5D 91 // GUID in RefShaderCache-PC-D3D-SM3.upk
01 00 00 00 // 1 entry follows
01 00 00 00 // 1 entry (or texture) follows
B0 03 00 00 // ExportTable; Texture2D'GEN_MaterialParents.Textures.Flags_DIF'
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 
E8 3F 89 40 // <<< The same for the most or all materials in given upk file.

// EW CHH_Flags	
<None> 
00 00 00 00 00 00 00 00 
01 00 00 00 // 1 entry follows
31 97 D4 1F A8 78 73 46 96 4C 32 1C 64 C7 5D 91 // <<< The same as in EU version
01 00 00 00 // 1 entry follows
01 00 00 00 // 1 entry (or texture) follows
C6 03 00 00 // ExportTable; Texture2D'GEN_MaterialParents.Textures.Flags_DIF'
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 
68 15 65 38 // <<< Different

GUID 31 97 D4 1F A8 78 73 46 96 4C 32 1C 64 C7 5D 91 is present in 1 chunk (material shader map) inside EW RefShaderCache-PC-D3D-SM3.upk.

 

CHH_Soldiers parent material:

// EU CHH_Soldiers
<None>
00 00 00 00 00 00 00 00 
01 00 00 00 // 1 entry follows
7C A9 4E 6C 1B DD 0B 47 A1 45 75 AB A2 38 CC 32 // GUID in RefShaderCache-PC-D3D-SM3.upk
01 00 00 00 // 1 entry follows
03 00 00 00 // 3 entries (or textures) follow
86 FF FF FF // ImportTable; Texture2D'GEN_MaterialParents.Textures.Default_Normal'
AF 03 00 00 // ExportTable; Texture2D'GEN_MaterialParents.Textures.Default_Spec'
87 FF FF FF // ImportTable; Texture2D'GEN_MaterialParents.Textures.Default_Diff'
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
03 00 00 00 // 3 entries follow
00 00 00 00 01 00 00 00 00 00 80 3F 00 00 80 3F // LinearColor?
00 00 00 00 02 00 00 00 00 00 80 3F 00 00 80 3F // LinearColor?
00 00 00 00 00 00 00 00 00 00 80 3F 00 00 80 3F // LinearColor?
E8 3F 89 40

// EW CHH_Soldiers
<None>
00 00 00 00 00 00 00 00 
01 00 00 00 // 1 entry follows
16 0C C4 39 3E 29 7F 47 9D EF 6C E1 AE 45 44 10 // <<< Different
01 00 00 00 // 1 entry follows
03 00 00 00 // 3 entries (or textures) follow
87 FF FF FF // ImportTable; Texture2D'GEN_MaterialParents.Textures.Default_Normal'
C5 03 00 00 // ExportTable; Texture2D'GEN_MaterialParents.Textures.Default_Spec'
88 FF FF FF // ImportTable; Texture2D'GEN_MaterialParents.Textures.Default_Diff'
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
03 00 00 00 // 3 entries follow
00 00 00 00 01 00 00 00 00 00 80 3F 00 00 80 3F // LinearColor?
00 00 00 00 02 00 00 00 00 00 80 3F 00 00 80 3F // LinearColor?
00 00 00 00 00 00 00 00 00 00 80 3F 00 00 80 3F // LinearColor?
68 15 65 38 // <<< Different

GUID 16 0C C4 39 3E 29 7F 47 9D EF 6C E1 AE 45 44 10 is in 13 chunks (material shader maps) inside EW RefShaderCache-PC-D3D-SM3.upk. Could be different maps for soldiers, SHIVs, weapons, cinematics, civilians, I don't know for sure atm.

 

 

What next?

 

I don't know, what communication is going on between the material and shader cache file, what is the logic. Does the mat querry the cache file, or does the cache file just assigns shader map to it? How does the cache file knows, which material needs shader map?

 

Maybe next step is to utilize UDK, create a package with a material, cook it, backup shader cache file, add a texture to the material, recook, compare cache files to find out where those chunks begin and end and then copy some chunks from EU to EW shader cache.

Edited June 30, 2017 by Drakous79

[Drakous79]

One more observation: in EU, DLCs had their own RefShaderCache-PC-D3D-SM3.upk.

Edited January 23, 2016 by Drakous79

[Zyxpsilon]

If only that could lead to finally fix the "Missing EU maps" (indirect solution by LiQuiD911 works partially) for proper integration into LW (even as a stand-alone external MOD) .. i'm all for it! Textures, materials & shaders are closely related while i still have to suspect the transition between EU & EW proved to be a bit more complex than expected for Firaxis devs.

 

:geek: Sorry i can't help out much about such coding stuff but.. i'm certainly wishing you the very best of luck.

[Drakous79]

Thank you Zyx, will do my best.

 

LiQuiD911 did great job wonders with maps, and pieces of his research are helping me to push it little further. Though I am not a coder myself, I like to poke things and watch what happens :smile:

 

Yesterday I managed to separate female's armor texture from male's. Under normal conditions males and females in EW share the same texture.

 

So I patched new image data, linked ChartexturesEU.tfc and nothing. Females had male texture.

Next I changed names of DIF, NRM and SPC texture. Males had female texture.

It turned out, that names of DIF, NRM, SPC texture AND a material instance constant have to be changed.

 

Enclosing mod file, that demonstrates the change. It patches female's kevlar diffuse texture with normal, so the change is clearly seen.

 

 

UPK_FILE=Soldier_FemaleKevlar_SF.upk


// Adds CharTexturesEU name to NameTable.

[ADD_NAME_ENTRY]
<%u 15> // String length
<%t "CharTexturesEU"> // ASCII null-terminated string
<%u 0x00000000> // Flags L
<%u 0x00070010> // Flags H


// Separates female's texture from male's.

RENAME=CHH_Lv1MedMale_MOD:CHH_Lv1MedGirl_MOD // This one is just for better orientation.
RENAME=MInst_Lv1MedMale1:MInst_Lv1MedGirl1
RENAME=Lv1MedMale_DIF:Lv1MedGirl_DIF
RENAME=Lv1MedMale_NRM:Lv1MedGirl_NRM
RENAME=Lv1MedMale_SPC:Lv1MedGirl_SPC


// Diffuse (colors) texture - changes TextureFileCacheName and patches new image data.

OBJECT=CHH_Lv1MedGirl_MOD.Textures.Lv1MedGirl_DIF:AUTO
[BEFORE_CODE]
<CharTextures>
[AFTER_CODE]
<CharTexturesEU>

REL_OFFSET=0x14d
[MODDED_CODE]
0C 00 00 00 // 12 mipmaps saved
// No high-res image
// 21 00 00 00, uncompressed size 0, compressed size -1, TFC offset -1, width, height
21 00 00 00 00 00 00 00 FF FF FF FF FF FF FF FF 00 08 00 00 00 08 00 00 // 2048x2048
// Image saved in TFC
// 11 00 00 00, uncompressed size, compressed size, TFC offset, width, height
11 00 00 00 00 00 08 00 A9 08 06 00 9C CC 77 05 00 04 00 00 00 04 00 00 // 1024x1024
11 00 00 00 00 00 02 00 17 A6 01 00 85 26 76 05 00 02 00 00 00 02 00 00 // 512x512
11 00 00 00 00 80 00 00 57 73 00 00 2E B3 75 05 00 01 00 00 00 01 00 00 // 256x256
11 00 00 00 00 20 00 00 32 1F 00 00 FC 93 75 05 80 00 00 00 80 00 00 00 // 128x128
// Image saved in UPK
// 00 00 00 00, uncompressed size = compressed size, UPK offset to the next byte, image data, width, height
// 64x64, 32x32, 16x16, 8x8, 4x4, 4x4, 4x4
00 00 00 00 00 08 00 00 00 08 00 00 51 F5 10 00 3E 8D 3F 7B BF 63 57 2B FF 84 BD 81 08 D5 FF 80 BD 9A 1E 75 FF A0 54 54 BE 85 3F 83 FF 55 AA 00 7F 75 9D 6A 82 55 56 AE 3E 82 1F 7D F7 60 A0 D5 9F 83 DF 74 AA A9 A8 AB 3F 7C 1D 72 A8 78 AA AA 1F 7C 1D 7A 02 2D 2A 2A 7F 84 7F 73 AB D0 DA A6 7E 9B FF 73 FD A1 29 25 FF 7B 3D 72 00 57 28 00 9F 83 DF 6C AA A8 6A EA 1F 84 BE 5C 00 80 03 09 1F 84 1F 74 AE 8E 89 29 9F 83 DF 7C AA AA AA 85 FC A1 BF 6C 5F FF FF F3 FF 83 DE 64 B5 DC 80 A0 9E A3 7F 6C F0 51 D1 D7 DF 84 7F 83 D5 00 FF A0 9F 94 5D 43 29 01 AA 88 FF 9C 3F 63 A0 BE FF 6F FF 7B 1C 62 00 00 02 01 9F 7B 1F 6C EE 92 58 52 3F 8C 7F 83 F5 7B A0 00 BF 7C DF 73 F5 8D 5A 57 7F 93 FF 6B 27 BF F7 75 5F 93 5F 64 F8 60 C2 8A 1F 8C BF 63 AA AF AD AF FF 83 7E 54 26 A4 28 98 9F 7C 3F 7B BB A2 E2 9E BF 84 FF 7B 67 59 67 F4 7E 92 FF 7B C9 B5 55 55 1E A3 1F 74 DE 72 E3 4F DF 9B DF 63 FA 6A E9 E1 FF 83 DF 6B 02 A0 C8 62 1F 94 1E 4B 2A CA F2 62 FF 7B 9E 62 7E 39 0A 00 FF 7B DE 53 02 00 A0 98 9F 83 DF 7C FA FF 54 02 1F 8C DF 6B 02 82 F5 25 BF 7C BF 73 55 F5 A5 0B 3F 84 FF 63 4F 6F 6E 8C 3F 8C 9F 7B FF EF 6E C8 3E 8D 7F 83 C2 DF D5 F5 FF 7B FD 43 90 B0 B0 28 DF 83 FE 64 88 88 40 A2 FF 7B 1E 6D 17 28 09 00 DF 83 1F 7C FF FF FF FF DE AB FF 63 47 E5 FD 51 DF 8B BF 6B 81 81 82 C2 DF 83 DF 6B 62 72 62 42 DF 93 DE 4B 42 42 CA 2A 3E A4 DF 6B FC F8 F8 FB 3F 7C 5F 6B A6 3E 8E 00 9F 93 3F 74 F0 5F 7B EB FE A3 1F 5C 2F 27 F5 FF 3E 93 1E 7D 56 B0 F6 BC BF 8C 5E 6B A0 75 7B 79 FF 93 DF 63 8E AD 0D 09 3F 94 5F 7C 25 AF FE 78 1F 8C 7E 5B 24 87 0D CA BF 7C 3F 7B D7 02 FF DE FE 7C 7F 7B FD DC C3 F8 1F 84 DF 73 AA 6A 6A 6A 5D EC FF 63 59 79 5F DF FE 84 DF 83 55 5F DC 2E FF 84 DF 7B 55 55 7F 80 DF 9B 9F 5C EA CB 8F 34 DF 9B 3E 4C A3 A1 AB AB 3F 83 9F 74 AF 2A BF D7 9E A3 DF 6B EB F0 58 5E 7E BC 1F 74 B5 2F AF F5 DF 84 FF 82 00 57 FF AA 9F 6C 3F 6B A0 F5 EB EA 1F 94 BF 6B 2D 2D 29 09 DF 9B 3F 74 EA E8 7E 5E BF 8B 7F 74 2F 7A C2 8A 3E AB 3F 6C 75 47 FD 7F 9F 7B BF 74 B4 B4 A9 A0 3E A4 DF 6B 7F 7F FF 3F 3E AC BF 63 3B 3F F6 67 FF 7B 7B 61 02 02 88 F5 FF 7B 1C 79 00 00 2A FF 5C BA 7F 5C 9F 9B FA F2 FF 8B 1F 6C 87 8E AE 94 1F 8C FE 63 DE D6 17 8E 5F 84 FF 6B 5E DE 5A F8 DF 9B 1F 74 AD 0F 0D 2F 5F 8B 3F 74 55 FE A8 DE 1F 8C 1F 5C DE DA B8 A0 5F 93 BF 6C 0B 2F FD 55 1F 83 9F 7C FA 80 82 55 DF 94 5A 51 AA 7A 1E 06 3C 91 1F 75 6A 2A D5 7F 19 A9 1F 7C 55 57 54 5B 1C DC FF 7B 95 95 95 95 DF A3 3E 5C 47 47 66 BE DE 91 DD 65 F7 55 5B 0B 5D 8E 3E 72 AF 2A 00 55 3B BE BF 73 76 7C DD DD 1F 9C FE 53 BC BC B4 B7 3F 83 DF 7C 77 FF AA 02 1F 75 BE 6A 00 0A FE 55 FE 8C 9E 82 00 00 AF F5 DF 8C 9F 73 B8 A0 5E 5F 5F 83 DF 7C FF FF 80 E8 9F 73 DE 6C 5F AA AA 0A 7F 83 BF 74 F5 AA 2A AA 1E AC 5B 23 87 8D 8D 8D 3F 9C FF 6B 4D 4D C9 49 5A D3 FF 53 F1 E1 E1 E1 BF C3 1F 5C BF FF FF 7F BC CA DF 6B DE 58 78 7E FF 83 7E 5B 24 0B 01 0A 7F 84 7F 73 8A 80 FA 5E 9F A4 DF 72 3F CA 7B 7B 5F 9C FC 3A A7 A7 85 89 5F 8B 5E 5C 0F D7 5E C2 1F 8B DF 7C 80 AB 55 F5 BF 8B FF 7C 28 00 F5 57 FE 9C 9F 8B 55 F5 2A AA 9F 8B 5E 7D 02 2A D5 56 BF 84 9F 73 F5 AF 02 60 BF 84 3F 6B 0A 00 70 B5 7F 9B 3C 3D 07 26 92 E2 FE 84 FD 61 C0 F2 37 00 5B DD 5F 53 F3 FA F6 FD 3F 84 9F 6B 6A DA 1A 02 FE A4 DF 5B E6 46 CE A8 3F 9C 9F 6B AD 2D 03 BF 3F 9C DE 53 FE FC F4 B4 9F 9B DE 5C E2 62 7B 6A 9F 8B 7E 75 88 8A 2B A5 FF 7B FE 6A 00 00 60 E0 5F 8C DE 72 AE 5E F5 2F 3F 8C 9E 72 2A 55 AA 00 5F 8C BE 7A 00 FD FF E0 DF 83 1E 6B A8 0A 37 2E FF 7B 3F 7B 25 00 5D B6 DE 82 FF 7B 55 55 EA F2 BF 9C 9F 6B 2F F7 A1 F1 BE 92 9F 74 7A FC E1 C5 1F 7C FF 83 A8 AA AA AA 9B 8E FF 7B 3D 81 FD 55 BE 95 FF 72 E8 FC FF 5F BF 7C FE 7A AA 55 57 8F 7E 8A 3F 74 56 AC AA 5F DE 8A 9F 7C 7F F2 FC 8A FF 7B 7C 6E B6 5E 20 00 3D 86 FF 7B 57 54 57 D5 5F 7C 3F 73 F0 55 AA AA DE 9B DF 73 0F 07 25 25 9F 93 BF 84 5E 00 AA FF 1F 9C BF 63 2E BE BF 37 9F 93 1F 5C EA 2C 3E 9C DF 9B 3F 6C F1 B8 F8 78 1F 8B BE 5D 8A 2E A2 0D FF 82 FE 74 1F 5A 8A 09 BE 92 BF 74 FE F0 C5 EE 3F 83 DF 74 AA 2A EE 52 DE 7A 7F 74 BA 78 FD D5 BD A5 5F 73 DF BD 43 57 7F 83 FD 5D 28 BF 40 00 BF 93 BE 5C 03 8A A9 E3 FF 8C FF 7A FF D5 E2 8A 5F 7C 5F 7B 2A 2B AA A9 3F 8C 1F 6C DE D2 FF 97 5F 94 3F 6C 0F 0F 3F 05 7F 94 1F 7C 5C 56 57 55 1F 8C FE 5B B6 16 B6 36 DE A3 3F 54 BE 9E 3F 3E 3E A4 1F 7C 5E 5C 5C 5C FF 7B 71 47 08 A0 40 00 9E 9A 3B 4F F8 82 0B 2D DE 92 9F 7C DD F5 FE E0 7F 84 1F 73 2A AA AA 2A 7F 83 1E 7D 57 0A AA 5F FD 75 1F 74 F7 05 BD 57 DD 85 3F 7C 75 00 E0 5D FE 8C 7F 7B 0A 5B FC 55 9F 84 1F 83 80 AA AA FD 1F 84 DF 73 5A FA 2A 27 FF 93 BE 53 9E 9E 9C 1C 1E AC DF 73 E5 E9 E1 E1 5F 8C FF 73 D5 D5 B5 2F BF 83 DE 5C 2A 96 AB 2E FE A3 3F 6C 37 17 97 37 1F 9C FF 73 5C 5C 54 D6 1F B4 FC 34 AA AA FA 6C 1F 7C 5A 25 24 38 20 30 DF 8B FF 63 60 40 43 C3 3D B5 FF 83 15 85 ED 79 7E AD DE 62 E3 5A D7 AF 7E 82 DF 7C EA A0 AB F5 BF 85 3F 83 BA FD FF 7B 9F 83 9D 5D 09 37 3E 28 FF 84 3F 83 FD BD 2D 2D 9F 82 1E 75 FE FF 5D 25 3E 8D 9F 73 D7 B5 0A D5 BF 93 BE 5C 02 83 A9 2A BF 93 BF 7B AB 2D 36 56 9F 93 DF 63 A6 B7 96 B8 3E AB FF 73 95 17 AD D5 BF 8C 7F 83 F5 FD AF 0B 1F 7C FC 79 D4 00 00 00 FF 83 BB 3A B0 B0 34 26 9F 9C DF 63 6B 6B 6A E0 BE AC 5F 7B E2 68 5C 54 DE 82 1F 7C F7 37 A0 5E 7F 8C 3F 73 FF F5 0F 0E 3F 83 9E 75 20 3A BD FF DF 84 9F 7B 5F FE E9 60 7F 8C 1F 7B 2D AF 2F 0B 5F 84 5F 73 9C B5 AF 80 DF 83 1F 7C FF FF FF FF DF 82 9F 7C BF FF FF FF BE 82 FF 7B 70 75 55 55 1F 83 7F 74 B2 09 77 5F FF 8C 1F 83 FF F5 AA F0 7F 84 1F 7B 56 AE AA AA 5F 95 1A 51 6A 6A B2 2E DF A5 3B 51 2D 8B A2 AA DF 62 3E 9D A9 AA AA AA FD B3 BF 64 B4 FC F3 83 9F 84 FF 6B D0 1A 0F C5 FE 92 7F 74 CA 2F 65 AB 5E 9C 5E 6B D1 53 52 5B 9F 7C 5F 7B 78 7B E5 73 5F 84 5F 83 09 0B AF 0D BF 83 7F 7C B6 BC A8 B6 FD 8D FF 8B 55 FC F9 F5 DF 7B FE 74 00 B5 55 FF FF A3 7E 44 AA BF BF 37 BF 9B FF 6B FF FF 7A 58 7F 84 5F 6B AA A0 2A 9E 5F 83 3E 75 2A 55 DF A0 3E 8D 3E 72 89 00 AA AA 7D BC 3F 7C 75 79 F9 39 1F 7C 5D 55 02 0C 3B BD 9C C3 FF 83 85 15 55 55 7E 95 3F 6B 77 F7 0F EE DF 8C 9E 72 AD F8 B8 AA 1E 85 FF 73 A1 60 7D 7D FF 82 9F 7C B5 AF 28 BE 5F 84 BE 82 0A 0B AD AA FF 83 1E 7D 27 02 08 00 7D 8A FF 83 79 05 81 55 FF 7B 9D 61 80 FA EA 00 7F 94 3E 63 B7 54 AD AA 9F 8C 9E 6A A0 7B AA AA 1F 8B BF 7C 57 22 AF FF DF 84 3D 72 F0 5F AF AA DF 83 7C 79 2A D5 00 00 3D 9A DF 74 2B AF 5F FF 1F 7C 1D 62 AD 3B 08 00 40 00 00 00 40 00 00 00 
00 00 00 00 00 02 00 00 00 02 00 00 69 FD 10 00 5F 8B 7F 74 8B BF 57 E8 BF 83 3E 7D 00 54 A8 AA 7F 94 7F 63 FF AD 81 FA 1F 7C FE 72 60 E0 A0 23 1F 83 3F 7C DC 7E 7A DD 5F 93 FF 73 BB D3 69 A5 DF 8B 5F 64 AA BE 9A EB 9F 83 3F 7C FF DD DD 59 5F 93 3F 74 43 2F 9F 5F DF 83 DF 73 8C EB 62 60 3F 8C 5F 6B 5B B6 86 8A DF 8B 1F 74 7F C7 47 E7 1F 94 1F 6C 7A F9 C1 C5 DF 93 FF 73 B5 B5 97 17 3F 8C BF 63 60 73 92 30 BF 83 9F 74 DA BE BA 92 1F 94 FF 73 4F 47 37 DB 9F 84 7E 7A AA 00 AA 55 9E 9A BF 6C FF FD FE FC 9F 8B 5F 64 8A CB F7 EA 3E A4 DF 7B 61 5B 59 59 DF 93 1F 6C 37 35 37 5E 3F 8C DE 6A 08 A8 6A 1A 9E 8A BF 74 FF FF F0 C7 5E AB 3F 6C DB DB 4F C7 BF 7B 7E 75 65 38 A8 00 9F 9C 1E 53 E8 BA 9A 9E BF 7C 3F 7B 02 57 FE 0B 1F 8B FF 84 D5 B8 AA F5 3E 7D DF 7B FF 55 D5 E0 1F 94 FD 43 12 1A 1A 8A 3E B4 DF 63 EF AF C7 FF DE 95 5F 73 F7 AF E2 7F 5F 7C 5F 7B CD A8 DF DD 7F 8B 9E 6D 28 BE 58 A0 9F 7C 1F 7B 7A DE F8 AA 1F 94 3F 73 AD EA 03 83 FF 8B 9F 73 BE 79 A2 79 3F 8B 5F 7C FF 78 9F 96 5F 8B 5F 7C F4 F1 C7 93 7F 7C 7F 7B D7 83 8A 0A 1E 7D BF 7B 6D D7 03 8A 7F 84 5F 7B DB 0A 88 FE 1F 84 DF 6B CA FA 7E 62 1F 94 1F 74 63 F1 5B B0 1F 8C 1F 6C FD 3D 3D 39 DF 8B FE 64 E8 68 A8 6A 3F 93 7E 5D AC 81 AF AE BE 9C 5F 7B 43 DA 78 56 1F 85 FF 7A AD EE FD 8B 9F 83 3E 75 09 CA 82 8B FF 84 BF 73 DF B4 F5 FF 5F 8B 5F 7C AB BD CF FF 9F 84 5F 7B EF FD D7 8A 9F 8C 9E 6A EA A2 6E AA DF 94 BE 52 AE 8D A3 A8 1E AC 3F 6C 7E DA D3 FF 3F 83 BF 7C B2 2B 5A FF 7F 83 5F 7C 61 CA 69 80 9F 84 FF 82 8A 0A 78 FA BF 74 1F 73 8A E0 F7 B7 1F 84 9F 73 82 B4 04 AA BF 7C 7E 7A 00 AA 56 AA 1E 8B 5F 6D BE DA 82 AE 20 00 00 00 20 00 00 00 
00 00 00 00 80 00 00 00 80 00 00 00 81 FF 10 00 9F 83 5F 7C 73 F6 A0 AF 3F 84 9F 73 6D F0 37 B3 BF 83 1F 74 8C 34 17 B9 3F 84 DF 6B AA AF A4 AC 9F 82 BF 7C FF AF 5F FB FF 83 FF 6B 28 F8 CC 34 3F 94 1F 6C 98 FE 6E E2 DF 93 FF 6B FA BE 37 BF FF 84 7F 7B FB 5C FF 2B BF 7C 7F 7B 7A E1 BF FF 9F 93 1F 74 FA FA DF D6 DF 83 7F 74 88 22 2E 9E 5F 8C 9F 7B 9C B6 78 BA 7F 83 7F 7C D9 EB 7A 3A 9F 83 5F 74 22 AA E5 A8 5F 84 7F 7B 1E 87 00 75 10 00 00 00 10 00 00 00 
00 00 00 00 20 00 00 00 20 00 00 00 19 00 11 00 3F 7C BF 7B 73 9F F6 B3 DF 83 3F 74 B2 EB 8C 3C 5F 84 BF 7B 64 F2 AF BE BF 83 1F 7C A8 57 6A ED 08 00 00 00 08 00 00 00 
00 00 00 00 08 00 00 00 08 00 00 00 51 00 11 00 3F 84 DF 7B D7 E7 FE DE 04 00 00 00 04 00 00 00 
00 00 00 00 08 00 00 00 08 00 00 00 71 00 11 00 1F 84 FF 7B DD EE DD EE 04 00 00 00 04 00 00 00 
00 00 00 00 08 00 00 00 08 00 00 00 91 00 11 00 FF 83 FF 7B FF FF FF FF 04 00 00 00 04 00 00 00 
// GUID
3D D5 3B F7 56 D3 E1 4B AF BE B1 6B 53 4E EC 17 

 

 

 

I still haven't verified, if mip maps stored in the UPK need correct bulk data offset or not. You know, 4 bytes pointing to UPK offset of the next byte. I saw violet armor with zoomed out camera, but it could be just higher res mip.

 

My plan is to create one colored (red and yellow) lower res mips and test, if they can be seen with incorrect UPK offsets. If not, then I'll correct offsets and re-check.

---

 

On shader's field not much changed. I've done quick and dirty job and patched EU serialized data at the end of EW file with zero results. I may have overlooked something.

 

Gonna install UDK today.

Edited January 26, 2016 by Drakous79

[Drakous79]

One more observation: in EU, DLCs had their own RefShaderCache-PC-D3D-SM3.upk.

 

I've compared DLC_PackIn, DLC_Day060 and CookedPCConsole shader cache files.

 

All three have 1563 names in NameTable.

 

DLC_PackIn added 3 armor decos, 3 hemets and 1 hair to the game. It doesn't need 1563 parameter names in shader cache, unless its cache overrides the one in CookedPCConsole.

 

This is not the way to go.

[tracktwo]

I've been following this with interest, but I'm a coder and not an artist and lots of the terminology is foreign to me. One thing that might end up helping a lot is the XCOM2 mod tools when they're available. Knowing how to add new materials in XCOM2 may make the process of adding them in 1 much simpler.

 

One thing to consider is that the more that can be done through standalone packages the better. This might not help with some mods where you want to tweak particular existing assets, but introducing brand new materials, textures, models, etc. might be feasible just by cooking new packages with those assets in the UDK. I had a lot of success with both the voice packs and custom flash UI simply by following the regular Unreal workflow instead of trying to patch existing UPKs or build the packages by hand from the ground up. Introducing new scaleform graphics packages was pretty straightforward, for example - just build the .swf and import it into the UDK and cook. But these don't involve any kind of global cache packages, which might be a bigger issue here, although the scaleform packages do include new textures.

[Drakous79]

Agreed tracktwo. Can't wait to try XCOM 2 SDK. From what I read on the internet, XCOM 2 runs on modified Unreal Engine 3.5, so I am very curious how it handles materials in mods, how and if are shader cache files used and updated.

 

I've tested new stand alone package with new model, material and textures year and half ago and everything I tried failed. New material never compiled. I'd like to know, how to make it to compile, or at least link it to some existing material or an entry in cache files. New packages are fine and easier to create, but sometimes it is feasible to patch small changes here and there.

 

Your advances with UI and flash + XCOM 2 and its overcustomization = me thinking about creating new XCOM 2 like customization UI for EW, but I don't have enough knowledge to do it the right way.

 

I remember your success with sound, that was fantastic. Have you had to patch sound file's footer with custom data to make it work? I remember something like that, or with licensee numbers. As a coder you can see patterns, where I see just bytes. Yes, byte view mode :smile:

 

Back on mats. In my opinion new material should compile upon startup and update the shader cache file or link self to it somehow. There are mods for other UE3 games and they work with new mats.

 

Going to take EU armor model and change its licensee number to EW now.

 

Edit: Not working, compile error.

Edited January 29, 2016 by Drakous79

[tracktwo]

I had to go dig through some old threads to find the answer to your questions, but I found it.

 

Originally, sound replacement didn't work right out of the box with voice packages created with the free UDK. The problem was the voice classes contain native c++ code to handle mapping the sound cues to the various events in the game, and the real voice packs created by Firaxis have extra native data blobs in them that the game knows how to process. Amineri discovered that she could just rewrite the native function that did that mapping in uscript, and this allowed us to load a bog standard upk created by the free UDK as a voice pack despite lacking the native data blob.

 

Even before I got to the point of using the UDK to create new voice packages I did some modding of existing sound packages to replace the .ogg file that was embedded in them. For example, I swapped out the memorial wall music with new music. This *did* require some hand editing of the package to fix up some offsets related to the new ogg file size. But that is ordinarily all handled by the upk as part of the package cooking and isn't needed when making entirely new packages. So this is somewhat reminiscent of swapping a material in an existing package vs creating new packages with new mats.

 

I'll take a look when I get a chance to see if there is a similar something going on here. If there is a native code blob in the upks containing these materials, any upk created by the free UDK will be missing that blob and the game will not work correctly. I can't say for sure that this is happening with materials or not, but its a start.

[Drakous79]

Thank you. I'll continue analyzing caches and comparing them with UDK creations to understand the process.

[Drakous79]

Made one little connection. At the end of material instance constant (MIC) are 16 byte blocks followed by 8 bytes. It is GUID (or hash) followed by parameter's name from NameTable. All these blocks together could be used to identify the correct spot in RefShaderCache-PC-D3D-SM3 (true).

// Similar data like static switches below are at offset 0xAB3CD0E of RefShaderCache-PC-D3D-SM3.upk, 
// but with different hex for names.

// EW Soldier_FemaleKevlar_SF.upk, CHH_Lv1MedMale_MOD.Materials.MInst_Lv1MedMale1
// This MIC has parent material GEN_MaterialParents.Master_Materials.CHH_Soldiers

<None>
00 00 00 00 00 00 00 00 

01 00 00 00 // 1 entry follows
7E E8 F6 6F 84 7A 2D 4D BA E6 58 F0 31 71 13 84 // GUID in RefShaderCache-PC-D3D-SM3; MIC
01 00 00 00 // 1 entry follows
06 00 00 00 // 6 entries (or textures) follow
87 FF FF FF // ImportTable; Texture2D'GEN_MaterialParents.Textures.Default_Normal'
C5 03 00 00 // ExportTable; Texture2D'GEN_MaterialParents.Textures.Default_Spec'
88 FF FF FF // ImportTable; Texture2D'GEN_MaterialParents.Textures.Default_Diff' 
AC 03 00 00 // ExportTable; Texture2D'CHH_Lv1MedMale_MOD.Textures.Lv1MedMale_NRM'
AD 03 00 00 // ExportTable; Texture2D'CHH_Lv1MedMale_MOD.Textures.Lv1MedMale_SPC
AB 03 00 00 // ExportTable; Texture2D'CHH_Lv1MedMale_MOD.Textures.Lv1MedMale_DIF'
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

03 00 00 00 // 3 entries follow
00 00 00 00 01 00 00 00 00 00 80 3F 00 00 80 3F // LinearColor?
00 00 00 00 02 00 00 00 00 00 80 3F 00 00 80 3F // LinearColor?
00 00 00 00 00 00 00 00 00 00 80 3F 00 00 80 3F // LinearColor?

86 C2 B5 00 // Unknown

16 0C C4 39 3E 29 7F 47 9D EF 6C E1 AE 45 44 10 // GUID in RefShaderCache-PC-D3D-SM3; the same as in parent material

04 00 00 00 // int StaticSwitchParameterList; 4 items

1F 05 00 00 00 00 00 00 // NameTable; parameter name "Reflect"; MaterialExpressionStaticSwitchParameter_15
00 00 00 00 // bool Value; false
00 00 00 00 // bool bOverride; false
74 DD D6 54 97 9A 1B 42 B2 E9 D7 EF AA 4F 79 87 // Expression GUID in parent material

01 00 00 00 00 00 00 00 // NameTable; parameter name "2 Color Tinted?"; MaterialExpressionStaticSwitchParameter_2
01 00 00 00 // bool Value; true
01 00 00 00 // bool bOverride; true
E0 BA 1D E9 4C 52 BC 47 B7 EE DB 43 7F 17 21 AA // Expression GUID in parent material

33 06 00 00 00 00 00 00 // NameTable; parameter name "Tinted?"; MaterialExpressionStaticSwitchParameter_3
00 00 00 00 // bool Value; false
00 00 00 00 // bool bOverride; false
AD DD 25 F2 51 B4 BA 4E BA A5 57 A5 A5 39 2D B6 // Expression GUID in parent material

5F 06 00 00 00 00 00 00 // NameTable; parameter name "Use Cube?"; MaterialExpressionStaticSwitchParameter_4
00 00 00 00 // bool Value; false
00 00 00 00 // bool bOverride; false
B3 B9 CE FD 56 9F 81 42 AB BD 6D E7 66 86 CF 86 // Expression GUID in parent material

00 00 00 00 
00 00 00 00 
00 00 00 00

Downside is the cache has different NameTable values. Must look for free hex editor with wild card search support.

 

Edit: MIC (MInst_Flags) of CHH_Flags ends with <None>, no additional data. May explain, why is this EU material working in EW.

 

And NameTable entries involvment explains, why copy pasting EU's RefShaderCache didn't work. EU or UDK created RefShaderCache entry would have to be tweaked for EW's RefShaderCache NameTable. Might be easier with UDK and pseudo code - add name entry, patch data.

 

Updating names is quite a hassle. When I've ported female skeletal mesh from EU to EW, 109 bone names and 2 material references had to be updated. Mod "DEV - Fix the girl" summary: the 1st try - an abomination, the 2nd try - a monster, the 3rd try - the monster with the correct texture. But there is covert ops model in the package :smile:

Edited July 20, 2017 by Drakous79