Inserting textures back to .tfc - possible?

[Amphibious]

As far as I understand only way to change texture is by using texmod. However the method has it's limitations and it doesn't run natively on linux.

 

Since it is possible to extract texture with umodel a viable alternative would be inserting hex sequence of altered texture back into the original .tfc file that would replace vanilla texture. However problem is that I don't know where exactly and in which compression format to insert it back in.

 

So far I know that textures are stored in 3 large .tfc files : chartexture, textures and textures_startup. Various .upk (like i.e. soldier_malekevlar_sf.upk) are linked to them and contain both offsets and compression type for textures contained. So when you run umodel it uses data from .upk to extract textures from .tfc file.

 

Upk files can be decompressed and it should be possible in theory to analyze them with hex editor to figure out offsets and compression type for desired texture. Once you have that data it would be a piece of cake to insert altered texture data into .tfc file.

 

So what do you think is that possible?

Edited August 4, 2014 by Amphibious

[wghost81]

It is not possible to insert textures back to tfc files, but it is possible to make your own tfc file. Search the forums for texture-related modding, especially posts by Drakous79. He made a big progress in that direction.

[Amphibious]

It is not possible to insert textures back to tfc files, but it is possible to make your own tfc file. Search the forums for texture-related modding, especially posts by Drakous79. He made a big progress in that direction.

 

If someone knows how tfc files are structered then I don't see how it would be easier to create them anew from start then to simply replace old texture data with altered one. Perhaps I am wrong but I think they were talking about making tpf files which are texmod package files not tfc files.

 

It is (must be) technically possible for sure but problem is we don't know positions (address) nor compression settings of particular textures.

 

Umodel reads them and export them fine from data located in various upks. However if we knew how it does the job exactly (instructions are contained it that upk) perhaps it would be possible to reverse it.

 

Then all one would need to do it to compress one of the extracted textures back. First vanilla that would help locate that particular tex code within .tfc files. After that it would be easy to replace that code with altered one.

[wghost81]

tfc files are basically concatenated raw compressed texture data. Each texture is compressed and all the textures are concatenated in one big tfc file.

 

Texture2D objects extract textures from tfc files by file name, file offset and data size. Small textures are packed directly into Texture2D objects (within upk files) and large textures are packed into tfc file.

 

To replace one texture inside tfc file you'll need to pack your texture with LZO packer (that's what XCOMLZO was originally written for). But there is no guarantee you texture will be the same size as an old one. In fact, you can be sure it will be different. If it has smaller size, you can rewrite old data with a new one, leaving some garbage, and update all the references to that texture with a new size. And there are a dozens of such references for one texture, especially for pawns. If a new texture is bigger... well, you out of luck, as you can't shift other textures inside tfc, because in this case you'll need to update every reference in every Texture2D object. You can append a new texture to the end of an old tfc file or, better, create a new tfc file, containing new textures only, and update relevant Texture2D objects with new filename, offset and size.

 

Texmod replaces textures in memory, it is easier than find and replace all the references in all the files.

 

Umodel searches for textures by analyzing all packages. It's a very complex utility, hope Gildor will release its source code as he promised, because it contains a tons of information on handling unreal packages and objects.

[Amphibious]

tfc files are basically concatenated raw compressed texture data. Each texture is compressed and all the textures are concatenated in one big tfc file.

 

Texture2D objects extract textures from tfc files by file name, file offset and data size. Small textures are packed directly into Texture2D objects (within upk files) and large textures are packed into tfc file.

 

To replace one texture inside tfc file you'll need to pack your texture with LZO packer (that's what XCOMLZO was originally written for). But there is no guarantee you texture will be the same size as an old one. In fact, you can be sure it will be different. If it has smaller size, you can rewrite old data with a new one, leaving some garbage, and update all the references to that texture with a new size. And there are a dozens of such references for one texture, especially for pawns. If a new texture is bigger... well, you out of luck, as you can't shift other textures inside tfc, because in this case you'll need to update every reference in every Texture2D object. You can append a new texture to the end of an old tfc file or, better, create a new tfc file, containing new textures only, and update relevant Texture2D objects with new filename, offset and size.

 

Texmod replaces textures in memory, it is easier than find and replace all the references in all the files.

 

Umodel searches for textures by analyzing all packages. It's a very complex utility, hope Gildor will release its source code as he promised, because it contains a tons of information on handling unreal packages and objects.

 

ok first two paragraphs i knew.

 

i overlooked that size will change when tex file is edited but still that's only limitations in a sense you can't have a larger file. i don't know how exactly compressor you mention works so it could be difficult to set it in a way to produce outcome of the specified lenght. however tex files themself can be manipulated in a number of ways to set their size as you wish.

 

i have been digging through upk to find a smaller files which contains tex since it easier to work with small files

 

so i found hair_malehair_b_sf.upk and decompressed it. . it has one 1 texture inside it. so i looked at its hex code and found an interesting segment that was sticking out:

 

 

48 61 69 72 73 74 79 6C 65 20 54 65 78 74 75 72 65 00 44 00 00 00 00 00 00 00 16 00 00 00 20 00 00 00 00 00 00 00 6C 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00 33 00 00 00 00 00 00 00 00 00 00 40 00 00 00 40 00 00 00 40 00 00 80 40 4F 00 00 00 00 00 00 00 43 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00 68 00 00 00 00 00 00 00 27 00 00 00 00 00 00 00 6C 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00 2C 00 00 00 00 00 00 00 6F 63 2F B6 0E 9B 6D 46 9F 26 D3 13 F3 D4 44 9B 3B 00 00 00 00 00 00 00 46 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 01 00 00 00 44 00 00 00 00 00 00 00 17 00 00 00 20 00 00 00 00 00 00 00 6C 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00 33 00 00 00 00 00 00 00 00 00 80 3F 00 00 80 3F 00 00 80 3F 00 00 80 3F 4F 00 00 00 00 00 00 00 43 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00 1C 00 00 00 00 00 00 00 27 00 00 00 00 00 00 00 6C 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00 2C 00 00 00 00 00 00 00 D2 94 F8 A3 3D 6F D9 4E 86 E9 BA 77 A2 33 57 6F 3B 00 00 00 00 00 00 00 46 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 01 00 00 00 21 00 00 00 00 00 00 00 6B 00 00 00 00 00 00 00 0D 00 00 00 00 00 00 00 09 00 00 00

 

try opening it in hex editor and you will see that it starts with "Hairstyle Texture" and after it was another similar readable label. Other upk I looked at soldier_malekevlar_sf.upk didn't have segments like this one.

 

Perhaps it couild be that this segment stores address and compressions settings of that particular piece of texture. It a fairly small segment which could be easier to analyze and figure out its structure.

[wghost81]

Search the forum, Drakous79 already did Texture2D format analysis. :wink:

 

Compressed data size will be different for different uncompressed images, even if they have the same resolution. And tfc files can contain compressed data only.

 

Compression type for tfc files is set somewhere else, not inside Texture2D objects. Texture2D objects hold info on pixel format only (DXT1, DXT5...).

 

Tfc reference has the following format: UncompressedSize, CompressedSize, TFCOffset, Width, Height.

 

For uncompressed images inside upk files format is: UncompressedSize, CompressedSize=UncompressedSize, UPKOffset, UncompressedData, Width, Height.

 

If you'll use FindObjectEntry utility on Texture2D object you'll get something like:

 

FindObjectEntry
Name to find: WP_PistolModern.Textures.T_PistolModern_DIF
Found Export Object:
0x00001617 (5655): Texture2D'WP_PistolModern.Textures.T_PistolModern_DIF'
	TypeRef: 0xFFFFFFA5 -> Texture2D
	ParentClassRef: 0x00000000 -> 
	OwnerRef: 0x0000049F -> Textures
	NameIdx: 0x000008B5 (Index) 0x00000000 (Numeric) -> T_PistolModern_DIF
	ArchetypeRef: 0x00000000 -> 
	ObjectFlagsH: 0x00000000
	ObjectFlagsL: 0x000F0004
		0x00000004: Public
		0x00010000: LoadForClient
		0x00020000: LoadForServer
		0x00040000: LoadForEdit
		0x00080000: Standalone
	SerialSize: 0x00000CFC (3324)
	SerialOffset: 0x015D0A05
	ExportFlags: 0x00000001
		0x00000001: ForcedExport
	NetObjectCount: 0
	GUID: 00000000000000000000000000000000
	Unknown1: 0x00000000
Attempting deserialization:
UObject:
	PrevObjRef = 0x0000003E -> AnimNotify_BreakWindow_2
UDefaultPropertiesList:
UDefaultProperty:
	NameIdx: 0x00000782 (Index) 0x00000000 (Numeric) -> SizeX
	TypeIdx: 0x0000041C (Index) 0x00000000 (Numeric) -> IntProperty
	PropertySize: 0x00000004
	ArrayIdx: 0x00000000
	Integer: 0x00000200 = 512
UDefaultProperty:
	NameIdx: 0x00000783 (Index) 0x00000000 (Numeric) -> SizeY
	TypeIdx: 0x0000041C (Index) 0x00000000 (Numeric) -> IntProperty
	PropertySize: 0x00000004
	ArrayIdx: 0x00000000
	Integer: 0x00000200 = 512
UDefaultProperty:
	NameIdx: 0x000005E9 (Index) 0x00000000 (Numeric) -> OriginalSizeX
	TypeIdx: 0x0000041C (Index) 0x00000000 (Numeric) -> IntProperty
	PropertySize: 0x00000004
	ArrayIdx: 0x00000000
	Integer: 0x00000200 = 512
UDefaultProperty:
	NameIdx: 0x000005EA (Index) 0x00000000 (Numeric) -> OriginalSizeY
	TypeIdx: 0x0000041C (Index) 0x00000000 (Numeric) -> IntProperty
	PropertySize: 0x00000004
	ArrayIdx: 0x00000000
	Integer: 0x00000200 = 512
UDefaultProperty:
	NameIdx: 0x00000319 (Index) 0x00000000 (Numeric) -> Format
	TypeIdx: 0x0000018A (Index) 0x00000000 (Numeric) -> ByteProperty
	PropertySize: 0x00000008
	ArrayIdx: 0x00000000
	InnerNameIdx: 0x000002CE (Index) 0x00000000 (Numeric) -> EPixelFormat
	Name: 0x00000683 (Index) 0x00000000 (Numeric) = PF_DXT1
UDefaultProperty:
	NameIdx: 0x000008FF (Index) 0x00000000 (Numeric) -> TextureFileCacheName
	TypeIdx: 0x000005A6 (Index) 0x00000000 (Numeric) -> NameProperty
	PropertySize: 0x00000008
	ArrayIdx: 0x00000000
	Name: 0x00000914 (Index) 0x00000000 (Numeric) = Textures
UDefaultProperty:
	NameIdx: 0x00000580 (Index) 0x00000000 (Numeric) -> MipTailBaseIdx
	TypeIdx: 0x0000041C (Index) 0x00000000 (Numeric) -> IntProperty
	PropertySize: 0x00000004
	ArrayIdx: 0x00000000
	Integer: 0x00000009 = 9
UDefaultProperty:
	NameIdx: 0x00000309 (Index) 0x00000000 (Numeric) -> FirstResourceMemMip
	TypeIdx: 0x0000041C (Index) 0x00000000 (Numeric) -> IntProperty
	PropertySize: 0x00000004
	ArrayIdx: 0x00000000
	Integer: 0x00000003 = 3
UDefaultProperty:
	NameIdx: 0x0000048A (Index) 0x00000000 (Numeric) -> LODGroup
	TypeIdx: 0x0000018A (Index) 0x00000000 (Numeric) -> ByteProperty
	PropertySize: 0x00000008
	ArrayIdx: 0x00000000
	InnerNameIdx: 0x00000901 (Index) 0x00000000 (Numeric) -> TextureGroup
	Name: 0x0000090B (Index) 0x00000000 (Numeric) = TEXTUREGROUP_Weapon
UDefaultProperty:
	NameIdx: 0x000005BB (Index) 0x00000000 (Numeric) -> None
UObjectUnknown:
	Object unknown, can't deserialize!

 

 

Here you can see tfc name, image size and pixel format data.

 

Replacing one texture for one Texture2D object is not a big problem. Big problem is there are many references to the same texture inside many packages. Finding and updating them all is a pain.

[Amphibious]

SO in a nutshell the problem is since it isn't possible for a new replacement texture to have the same compressed and uncompressed size as vanilla texture it would be neccessary to alter all the references within all the upks to make it work. And that is a real hassle.

 

Am I getting it right?

 

 

edit: just one more thing there should be uncompressed small textures inside upk files right? so if i run umodel on upk without neccessary tfc files provided this is the image that gets extracted to texture2D folder. is that right?

 

However when I open it with hex editor and serch for parts of the image hex code in original uncompressed upk they are nowhere to be found.

Edited August 4, 2014 by Amphibious

[Amineri]

As wghost mentioned, some 2DTextures are stored directly within the package files, and are not redirected to lookups in a tfc. Generally these seem to be the GFX Flash based textures, not those that are wrapped around 3D models (although technically the Flash UI elements can be placed onto a 3D object, as is demonstrated in-game).

 

So far I've had success doing the following :

 

1) Replacing the strategy-game textures for flags. These are located in the UICollection_Common_SF.upk. These particular 2DTextures were packed as DXT1 format (uncompressed) image data. Since it is uncompressed I was able to perform a same-size byte swap to replace the Columbia Flag image with Honduras and the Turkey Flag image with Bosnia-Herzegovina. The reason we did this is that the existing armor models contain those two flags.

 

2) Adding new 2D Inventory images. I did this by using the September 2011 version of Unreal Development Kit to import BMP and PNG images into a new package file. Import setting used the TEXTUREGROUP_UI (to prevent low res version from being used). Images are referenced internally via strings such as "img:///LongWar.InventoryImages.Inv_LaserMachinePistol". The vanilla image paths for inventory images are all defined in the UIUtilities class, for example "img:///UILibrary_StrategyImages.InventoryIcons.Inv_Pistol". The vanilla images are pulled from UICollection_Strategy_SF.upk. The package file doesn't have to referenced in the vanilla imagepaths because it was cooked as a seek-free package. In contrast the LongWar.upk package we've made with our new inventory images aren't cooked.

 

3) Adding new 2D weapon panel images. This required XMTS to make some alterations to how the Flash UI code handles images, as there are two different methods of binding unreal 2DTextures to make the usable by the Flash sprites/actionscript. I'd have to defer to XMTS as to what alterations he made, but they allowed passed full unreal image paths as in #2 to be used.

 

 

As far as replacing textures on 3D models, so far I haven't made any significant progress.

[wghost81]

SO in a nutshell the problem is since it isn't possible for a new replacement texture to have the same compressed and uncompressed size as vanilla texture it would be neccessary to alter all the references within all the upks to make it work. And that is a real hassle.

Yes. If the replacement compressed texture was exactly the same size as original one — there wouldn't be any problems. But now replacing textures means updating all the references as well. It's doable, but requires more research.

edit: just one more thing there should be uncompressed small textures inside upk files right? so if i run umodel on upk without neccessary tfc files provided this is the image that gets extracted to texture2D folder. is that right?

However when I open it with hex editor and serch for parts of the image hex code in original uncompressed upk they are nowhere to be found.

They are in DXT1 format and umodel probably converts those to something else while exporting. Furthermore, packed textures are raw data without a header and umodel adds a header with palette and other info for graphical viewer.

 

Don't get me wrong, we can replace textures without texmod:

http://i.imgur.com/3t6fRgI.jpg

 

It's just not that simple like replacing the image data only. There is no Texmod for Linux yet, but may be it's time we create one. :smile:

Edited August 5, 2014 by wghost81

[Drakous79]

Hi guys :smile:

 

I've started to summarize all info few months ago, but failed to finish it in time. Will do my best to post it asap.

 

Appending textures to TFC files bigger than 500 MB is not the best idea, because of an amount of data, that would have to be downloaded in a case of new patch. Better is to change TextureFileCacheName property from NameProperty to StrProperty. Such change results in new and custom TFC file with name limited to 3 characters. With an addition of new name into UPK's nametable, or using some existing entry, all is needed is to redirect NameProperty to new one.

 

Another thing I haven't found out is, if TFC files can contain uncompressed textures and how to write such info into UPK files.

 

I think used compression is LZO-1x. Hope my memory is right - please correct me, if not.

 

 

Texture2D layout

 

4 bytes - unknown (0xffffffff for UI images)

 

Next is a list of texture's properties written as a sequence of name table indexes followed by values. Please see Default properties on pages 14-17 of XCOM EU/EW UPK file format described by wghost81 (available in miscellaneous files of UPKUtils) for details.

 

Let's look on an example: Weapon_Pistol_SF.upk, WP_PistolModern.Textures.T_PistolModern_DIF.

• SizeX=512 (width [pixels])
• SizeY=512 (height [pixels])
• OriginalSizeX=512 (width of source image [pixels])
• OriginalSizeY=512 (height of source image [pixels])
• Format=EPixelFormat.PF_DXT1 (format)
• TextureFileCacheName=Textures (name of texture file cache)
• MipTailBaseIdx=9
• FirstResourceMemMip=3
• LODGroup=TextureGroup.TEXTUREGROUP_Weapon (texture group the texture belongs to)

 

SizeX=512 would be:

8 bytes - 2B 05 00 00 00 00 00 00 - SizeX (NameIdx; property name)

8 bytes - 1A 03 00 00 00 00 00 00 - IntProperty (TypeIdx; property type name)

4 bytes - 04 00 00 00 - 4 (PropertySize; property size in bytes)

4 bytes - 00 00 00 00 - 0 (ArrayIdx; zero for non-array types)

4 bytes - 00 02 00 00 - 256 (Value; Integer)

 

 

After the list of texture's properties is ... ehm, here is the point, when I ran out of time :sad: but there should be 16 zero bytes and then this: UPK_Texture2D.pdf (Dropbox).

 

 

Texture's footer contains texture's ID and some unknown bytes.

 

78 5F E4 54 C0 D0 09 42 9A 6B 4C 3C 30 D2 04 3D - ID, the same as seen in GlobalPersistentCookerData.upk

00 00 00 00 01 00 00 00 00 00 00 00 - unknown, but seems to be the same for all model textures I looked at.

 

Btw I deleted GlobalPersistentCookerData.upk once and the game runs without it :smile:

 

 

---

There is PCConsoleTOC.txt file containing file sizes. It may be important for game's memory management, but it may also serve for patching purposes. No progress made on this one.

 

---

To create a texture, extract something made with Texmod, alter it, convert it to DDS (DXT1 or 5), delete DDS header (first 128 bytes), compress it with wghost's XCOMLZO (found in UPKUtils), put the result into custom TFC file and update its references in UPK :smile:

 

Usually, all mipmaps 128 px and higher are stored in TFC. Lower resolutions are stored in UPK.

 

Also as wghost mentioned, texture's info should be updated in all UPKs, because of game's cache.

 

---

I know we should be able to shift TFC offsets and make snow maps or create additional armor decos with altered textures.

Edited August 5, 2014 by Drakous79