XCOM EU Texture modding without texmod

[wghost81]

It's been a while :smile:

I was finally able to finish a project started a very-very long time ago by Drakous79. Thanks to the people from Batman modding community who approached me with the problem of texture modding in that game. I guess, 5 years away give you a broad enough perspective to realize what you've been doing wrong :smile:.

 

First off, Texture2D export data format. UE3 Texture2D object inheritance looks like this:

Object <- Surface <- Texture <- Texture2D

Its serialized data start with PrevObjectRef (same as everything else) followed by DefaultPropertiesList (see my old docs on upk format). Then come its own data as follows:

an empty bulk data chunk (12 zero bytes + 4 bytes of absolute file offset pointing to the MipMapCount)
MipMapCount (4 bytes)
serialized MipMaps array of MipMapCount elements
unknown data

Unknown data seems to be just a bunch of memory variables someone had forgotten to exclude from serialization.

 

MipMapCount should correspond to MipTailBaseIdx property defined in DefaultPropertiesList:

MipTailBaseIdx = MipMapCount - 1

MipMaps are BulkData with 8 additional bytes of SizeX and SizeY:

4 bytes of Flags
4 bytes of ElementCount
4 bytes of BulkDataSizeOnDisk
4 bytes of BulkDataOffsetInFile
BulkDataSizeOnDisk bytes of image data (if Flags do not have StoredInSeparateFile or StoredAsSeparateData bit set)
4 bytes of SizeX
4 bytes of SizeY

Possible Flags:

StoredInSeparateFile = 0x00000001
StoredAsSeparateData = 0x00000040
EmptyData = 0x00000020
CompressedZlib = 0x00000002
CompressedLzo = 0x00000010
CompressedLzx = 0x00000080

If StoredInSeparateFile bit is set, the TextureFileCacheName property defined in DefaultPropertiesList is used to obtain the name of the tfc file (texture cache file) - usually it's 'Textures'. Furthermore, externally stored mipmaps are compressed (usually lzo), so expect CompressedLzo bit being set too. For such mipmaps ElementCount corresponds to uncompressed size, BulkDataSizeOnDisk corresponds to compressed size and BulkDataOffsetInFile corresponds to Textures.tfc absolute offset. SizeX and SizeY contain corresponding image dimensions.

 

If StoredAsSeparateData bit is set, the image data are stored in the same package, but outside of this object serialized data.

 

If EmptyData bit is set, both BulkDataSizeOnDisk and BulkDataOffsetInFile are set to 0xFFFFFFFF and ElementCount is set to zero.

 

There are three interesting parameters defined in DefaultPropertiesList: Format, LODBias and NeverStream. Format is always present and defines texture block compression method. XCOM mostly uses BC1 and BC3 (DXT1/DXT5). LODBias is... well... LOD bias. It defines the number of textures to drop, effectively allowing to limit texture resolution. More info on textures and their properties can be found here and here. For modding purposes it's important to know that for, say, 2k texture you should have 12 mips defined, but LODBias set to 2 will limit the resolution to 512 and cause the first two mips to be dropped (the corresponding BulkData will be empty). Adding those mips back won't do anything unless you also mod LODBias to zero. Be careful here, though, as the engine calculates the final LODBias as a sum of several parameters (see the links above), so do not assume the LODBias value based on Texture2D properties only! NeverStream boolean set to true forbids the engine from streaming the texture out of memory. Mipmaps for such textures are usually fully embedded in a startup package (all the mipmaps for all the LOD levels).

 

So, with this knowledge in mind, what's the problem of modding textures? Well, it's how the cooking is done.

 

The most frequently used textures are not streamable and embedded into just one startup package. Those are easy enough to mod by just replacing their mipmap data. But lots of textures (map tiles, for example) end up being cooked into a dozen of different packages with low-res mips being embedded directly (usually up to 512) and hi-res ones being kept in an external tfc (Textures.tfc). It's done to speed up the loading process. So if we want to alter such a texture, we need to alter all the packages it resides in, plus its Textures.tfc data. Which is... quite a task at a first glance.

 

And this is where we made a mistake 5 years ago: making a PatcherGUI patch file to mod just one texture is a pure pain and it becomes unrealistic if we want to change dozens of textures. And no, Drakous79, my friend, upk lists won't help here ;) What we need is a completely new tool.

 

On the paper the process is simple enough: take an existing texture, export it to a dds file with all the mipmaps, edit and re-import. We can move/resize export object data if needed, we can change default properties, we can even alter the TextureFileCacheName to point at something else (Texture2D is an obvious choice as the property is a NameProperty and thus has to be present in the name table). But we also need to be sure we do it for all the packages that use this texture.

 

Currently, I'm working on a set of tools that allow mass exporting/importing of textures. The first one, ExportTexturesToDDS, is 99.9% ready and can export individual textures or the whole game into dds file(s) creating an inventory file along the way. The inventory file is a simple csv (comma-separated values) file that holds the full names of all the textures found with the list of packages they were found in. Using this list another tool, ImportTexturesFromDDS, can take all the textures you feed it and import them back into all the corresponding packages. At this point ImportTexturesFromDDS can already work with individual textures and mass-import is what I'm working on.

 

Since we can potentially go to higher resolutions, re-importing textures into the existing Textures.tfc doesn't seem to be the way here. Using a new custom tfc file is a better solution. But the problem is that we're working with cooked packages and while modding and testing stuff we're doing it one texture at a time, which complicates things due to Textures.tfc being just a bunch of "dumb" data with no additional information on what they belong to. To address the issue I came up with a custom tfc format that stores metadata on embedded mipmaps in a separate field.

 

Vanilla Textures.tfc format:

C1 83 2A 9E "magic" followed by lzo compressed blocks
alignment bytes (zero-padding, not used by xcom, used by batman)
C1 83 2A 9E "magic" ... etc

Custom Texture2D.tfc format:

2B 42 91 53 "magic"
4 bytes of BlockOffset
4 bytes of BlockSize
4 bytes of NextBlockOffset
a list of inventory entries
C1 83 2A 9E "magic" followed by lzo compressed blocks
...

Each inventory entry has the following data:

4 bytes of BulkDataSizeOnDisk
4 bytes of BulkDataOffsetInFile
4 bytes of ObjectNameLength
ObjectNameLength bytes of FullObjectName

Max block size is 0x8000. When the limit is reached, the tool makes another block and writes its offset to the NextBlockOffset field of the previous block.

 

The resulting macro-structure of a custom tfc looks like this:

[metadata block #1]
[compressed mipmap #1]
[compressed mipmap #2]
...
[metadata block #2]
[compressed mipmap #N]...

Metadata are ignored by the game itself, but are used by the importer to avoid hi-res texture duplication when importing a new texture on a per-texture per-package basis.

 

That's it. I can't promise I'm back, but I will sure finish my work on texture exporting/importing. Linux/Mac users will finally be able to play with military retexture pack :)

[dubiousintent]

Wow! :ohmy: I did not expect to see you revisiting this old chestnut of a game. (I just drop by to see that someone directs the lost souls to the wiki, myself.)

 

Impressive work, as always. Guess I'll have to write up one more article when you get something released.

 

-Dubious-

[wghost81]

He-he :) Didn't expect that myself :) But here I am, back to coding upk stuff. Thanks for keeping the forums alive, I miss this place and the good old times.

[wghost81]

A tutorial on how to mod the textures without texmod with the links to the necessary tools. Work in progress, but at this point it can be considered fairly stable. I will update and link github repositories later.

 

Export procedure was tested with both XCOM:EW and Batman: Arkham Asylum. Import procedure was tested with Asylum only.

[wghost81]

github repo: https://github.com/wghost/UPKUtils

[dubiousintent]

Published article "Modifying Textures without TexMod - XCOM:EU 2012". Let me know if there are any edits needed.

 

-Dubious-

[wghost81]

Great :thumbsup:

 

dubiousintent, let me use this opportunity to finally say that you are a pillar of this community. The wiki is a massive work and it's useful not only for XCOM modders but for all the UE3 modders out there. So, from the bottom of my heart - thank you for all these years of making sense of our random posts!

[dubiousintent]

Thank YOU! That is most gratifying to hear, but all my efforts would be worthless without the dedicated investigations of the rest of the community providing the raw grist of the wiki articles. I'm happy to have been able to contribute in some small way.

 

Best of luck in your current endeavors.

 

-Dubious-

[EvilMV]

Hi, All!

 

I used wghost81 tools to try modding XCom:EW. I made the first attempt to convert well-known Camo mod to not use TexMod. Unfortunately attempt isn't fully successful. I'll share my experience here: maybe some advice will appear or my work will inspire someone to help/continue.

 

What I did.

1. Unpacked TPF files of the Camo mod.

2. Renamed *.dds files to game names. For that I had to extract game textures and search source textures in extracted files for some days.

* There was one inconvenience here. Helmet modifications from the Camo mod have separate textures with lower resolutions (author hooked mip-map calls separately?). So unmatching textures I had to discard leaving highest res texture.

3. Imported textures to game using inventory.csv generated on extraction stage.

 

Results:

- generally there were no errors on import and tool generated in output folder Texture2D.tfc of size ~190M and 194 UPK files referencing changed resources

- game starts without errors after overwriting original UPK files with those ones

- I have camo textures on armor and helmets visible

However there are issues and funny behavior:

- when giving armor or weapon to the soldier there is visible trnsition until camo variant is rendered: basic->rainbow->camo

- actually all textures are looking like correct from the first sight: tey are camo textures. But they are very shiny on edges - significant bloom effect and white light near edges

- there is some pixelated noise on textures that reveals most significantly on weapon textures under certain angles

- some helmets have rainbow parts

- grapple hook on Kestrel armor is rainbow (I think it is grapple - on the left forearm, right?)

 

So, if someone have ideas where to go further from this point, it would be very good.

 

 

 

 

 

 

 

UPD: I think I've found the reason of a problem, at least for a hook. Need to review import logs more carefully.

Texture2D object found: WP_GrappleHook.Textures.GrappleHook_SPC
PixelFormat mismatch! Old PixelFormat = PF_DXT1

UPD2: Fixed pixel format for grapple hook. No rainbow texture but black.

Tried to do following on clean game:

- exported textures

- imported just Grapple Hook original, untouched textures (DIF, SPC ,NRM)

 

In game grapple hook on left forearm is black. Armors are camo but all sparking bright and pixelaed on the edges... So it looks like something is wrong,

 

On some replacement I see no issues. For example, no problems with Alloy Cannon.

 

UPD3: if do export/import/export for some texture, last exported differs from source by one byte on offset 0x00018. In source texture we have 00 here, but after import followed by export we have 01 in this byte. I believe this is volume texture depth, right? But I can't say anything about issues possibility if this depth is wrongly set to 1 (equivalent to setting DDS_HEADER_FLAGS_VOLUME?) instead of 0.

 

Dunno what to do next. Any advice where to look?

Edited July 12, 2020 by EvilMV

[Asesinamen]

So my objective is to replace the soldier class icons from LW with coloured ones in the texture files, as I've grown weary of using texmod (and also as a learning experience towards other replacements).

 

My main problem is that I cannot get ExportTexturestoDDS (gotten from the link in here) to unpack from LongWar.upk (in XEW\XComGame\CookedPCConsole), where I suspect LW's classes' icons to be (UICollection_Strategy_SF.upk only has the four vanilla classes', and replacing those doesn't replace them in-game when using LW even if four of the LW classes use vanilla classes' icons, so I suspect they use copies of them from LongWar.upk too).

 

The error when trying to unpack LongWar.upk is:

Texture2D object found: ClassIcons.archer_mec
terminate called after throwing an instance of 'std::length_error'
  what():  vector::reserve

Any ideas?