UPK file format

wghost81 · January 13, 2014

As I wrote before, default properties are not objects!

bPsiGift object, which is referenced through ExportTable, is an UByteProperty object. It inherits from UObject:

UByteProperty -> UProperty -> UField -> UObject

It's serialized data contain all the fields of UObject, followed by the fields of UField and then by UProperty and UByteProperty. Those data are used by the engine to reconstruct bPsiGift as an object.

But data in UDefaultProperty are not objects! Sorry, I didn't explained it better in the document, but default properties are a pain and I still can't deserialize all of them properly.

Now, default properties, which are contained inside corresponding UObject and UScriptStruct list are not objects, they are lists of initializers, which follow UDefaultProperty data format, described in Core types document section. For booleans format look like: 8 (NameIdx) + 8 (TypeIdx) + 4 (zeros) + 4 (zeros) + 1 (0 or 1).

wghost81 · January 13, 2014

While trying to prevent crashes and infinite loops caused by bad data, I accidentally mixed up UScriptStruct default properties deserialization. Sorry. Will update the code and utilities. This is purely info thing, so patcher functionality is not affected.

Here is correct output for TPsiTrainee and bPsiGift:

FindObjectEntry
Name to find: XGFacility_PsiLabs.TPsiTrainee.bPsiGift
Found Export Object:
0x00003877 (14455): BoolProperty'XGFacility_PsiLabs.TPsiTrainee.bPsiGift'
	TypeRef: 0xFFFFFEF2 -> BoolProperty
	ParentClassRef: 0x00000000 -> 
	OwnerRef: 0x0000387A -> TPsiTrainee
	NameIdx: 0x000004AF (Index) 0x00000000 (Numeric) -> bPsiGift
	ArchetypeRef: 0x00000000 -> 
	ObjectFlagsH: 0x00000000
	ObjectFlagsL: 0x00070004
		0x00000004: Public
		0x00010000: LoadForClient
		0x00020000: LoadForServer
		0x00040000: LoadForEdit
	SerialSize: 0x00000028 (40)
	SerialOffset: 0x0038BA87
	ExportFlags: 0x00000000
	NetObjectCount: 0
	GUID: 00000000000000000000000000000000
	Unknown1: 0x00000000
Attempting deserialization:
UObject:
	PrevObjRef = 0x00003876 -> m_arrCompleted
UDefaultPropertiesList:
	NameIdx: 0x00002594 (Index) 0x00000000 (Numeric) -> None
UField:
	NextRef = 0x00000000 -> 
UProperty:
	ArrayDim = 0x0001 (1)
	ElementSize = 0x0000 (0)
	PropertyFlagsL = 0x00000000
	PropertyFlagsH = 0x00000000
	CategoryIndex = 0x00002594 (Index) 0x00000000 (Numeric) -> None
	ArrayEnumRef = 0x00000000 -> 

FindObjectEntry
Name to find: XGFacility_PsiLabs.TPsiTrainee
Found Export Object:
0x0000387A (14458): ScriptStruct'XGFacility_PsiLabs.TPsiTrainee'
	TypeRef: 0xFFFFFEE1 -> ScriptStruct
	ParentClassRef: 0x00000000 -> 
	OwnerRef: 0x000038C5 -> XGFacility_PsiLabs
	NameIdx: 0x00003118 (Index) 0x00000000 (Numeric) -> TPsiTrainee
	ArchetypeRef: 0x00000000 -> 
	ObjectFlagsH: 0x00000000
	ObjectFlagsL: 0x00070004
		0x00000004: Public
		0x00010000: LoadForClient
		0x00020000: LoadForServer
		0x00040000: LoadForEdit
	SerialSize: 0x0000008D (141)
	SerialOffset: 0x0038BB03
	ExportFlags: 0x00000000
	NetObjectCount: 0
	GUID: 00000000000000000000000000000000
	Unknown1: 0x00000000
Attempting deserialization:
UObject:
	PrevObjRef = 0x00003879 -> kSoldier
UDefaultPropertiesList:
	NameIdx: 0x00002594 (Index) 0x00000000 (Numeric) -> None
UField:
	NextRef = 0x00000000 -> 
	ParentRef = 0x00000000 -> 
UStruct:
	ScriptTextRef = 0x00000000 -> 
	FirstChildRef = 0x00003879 -> kSoldier
	CppTextRef = 0x00000000 -> 
	Line = 0x00000000
	TextPos = 0x00000000
	ScriptMemorySize = 0x00000000
	ScriptSerialSize = 0x00000000
	Script decompiler is not implemented!
UScriptStruct:
	StructFlags = 0x00000000
UDefaultPropertiesList:
	NameIdx: 0x00001A0F (Index) 0x00000000 (Numeric) -> kSoldier
	TypeIdx: 0x000025D2 (Index) 0x00000000 (Numeric) -> ObjectProperty
	PropertySize: 0x00000004
	ArrayIdx: 0x00000000
	Object: 0x00000000 = none
	NameIdx: 0x0000159C (Index) 0x00000000 (Numeric) -> iHoursLeft
	TypeIdx: 0x000016A3 (Index) 0x00000000 (Numeric) -> IntProperty
	PropertySize: 0x00000004
	ArrayIdx: 0x00000000
	Integer: 0x00000000 = 0
	NameIdx: 0x000004AF (Index) 0x00000000 (Numeric) -> bPsiGift
	TypeIdx: 0x00000495 (Index) 0x00000000 (Numeric) -> BoolProperty
	PropertySize: 0x00000000
	ArrayIdx: 0x00000000
	Boolean value: 0x00 = false
	NameIdx: 0x00002594 (Index) 0x00000000 (Numeric) -> None

Edited January 13, 2014 by wghost81

Amineri · January 13, 2014

Sorry if it seemed like I was implying that DefaultProperties were objects ... I realize that they aren't :)

The bPsiGift variable I didn't have any trouble changing the type of (starting to get comfortable with that), but figuring out what was going on with the Default Properties (and the bPsiGift default property in particular) within the TPsiTrainee object is something I'm not quite as familiar with, and unfortunately your latest rev of the UPK Format doc was a little confusing to me on the subject.

Looking at the output above (combined with what I gleaned from both digging through the upk via UE Explorer and the UPK Format doc) it makes a bit more sense.

I'm not sure how far you want to go with your UPK Format document, but perhaps including some examples (as above) would help with reader comprehension.

wghost81 · January 13, 2014

Updated UPK Utils with FindObjectEntry fix.

Amineri, I want to make upk format document useful for the others. But sometimes it is difficult for me to express certain concepts, as English is not my first language. Don't hesitate to point me to my typos, grammar errors and especially confusing descriptions. I will try to fix it.

I will sure put some examples of manual data deserialization. Reflection and serialization concepts are not easy, especially for non-programmers, I had troubles understanding them myself. So I will try to explain it better. I'm a university teacher, after all. :smile: That's my job to make people understand what I'm saying. :smile:

EDIT: updated package format document in misc section of UPK Utils nexus downloads.

Edited January 13, 2014 by wghost81

wghost81 · January 21, 2014

Updated package format document in misc section of UPK Utils nexus downloads. Some minor info on HasStack and Default Properties.

Amineri · February 16, 2014

Using wghost's info and a bit of my own creative exploration, I've figured out how to (in practice) expand the size of an enum in an existing upk using UPKmodder.

It turns out that this is necessary as the engine has a distressing tendency to clamp variables to live within the defined range.

The motivation for this is in Long War EW, where we plan to expand the number of UFO types from the vanilla 9 to 14. The new UFOs will share graphics and maps with 5 of the existing UFOs, but will have different stats (for example, a Scout and a Fighter will have different stats in the air, but have the same hull and use the same maps). Unfortunately, we found that when attempting to create AI objectives using the new ship enums that the value would be clamped to within the given range defined by XGGameData.EShipType:

enum EShipType
{
    eShip_None,
    eShip_Interceptor,
    eShip_Skyranger,
    eShip_Firestorm,
    eShip_UFOSmallScout,
    eShip_UFOLargeScout,
    eShip_UFOAbductor,
    eShip_UFOSupply,
    eShip_UFOBattle,
    eShip_UFOEthereal,
    eShip_MAX
};

The format of an enum is actually quite simple.

It starts with an enum header consisting of five 4-byte integers (functions start with twelve 4-byte integers in their header). The only one that needs changing is the 5th byte which is the length of the enum.

For EShipType, this is:

7C 2E 00 00 76 5F 00 00 00 00 00 00 7C 2E 00 00 
0B 00 00 00 // length

Following this header are a variable size list (of the defined length) of 8-byte namelist references. For EShipType, this consists of :

54 2D 00 00 00 00 00 00 // eShip_None
52 2D 00 00 00 00 00 00 // eShip_Interceptor
55 2D 00 00 00 00 00 00 // eShip_Skyranger
51 2D 00 00 00 00 00 00 // eShip_Firestorm
5A 2D 00 00 00 00 00 00 // eShip_UFOSmallScout
59 2D 00 00 00 00 00 00 // eShip_UFOLargeScout
56 2D 00 00 00 00 00 00 // eShip_UFOAbductor
5B 2D 00 00 00 00 00 00 // eShip_UFOSupply
57 2D 00 00 00 00 00 00 // eShip_UFOBattle
58 2D 00 00 00 00 00 00 // eShip_UFOEthereal
53 2D 00 00 00 00 00 00 // eShip_MAX

(I've added the comments to improve clarity)

To alter the enum I simply resized the function (using exactly the same methods as I do to increase the size of a function), and inserted additional 8-byte namelist references, as well as changing the length to match :

54 2D 00 00 00 00 00 00 // eShip_None
52 2D 00 00 00 00 00 00 // eShip_Interceptor
55 2D 00 00 00 00 00 00 // eShip_Skyranger
51 2D 00 00 00 00 00 00 // eShip_Firestorm
5A 2D 00 00 00 00 00 00 // eShip_UFOSmallScout
59 2D 00 00 00 00 00 00 // eShip_UFOLargeScout
56 2D 00 00 00 00 00 00 // eShip_UFOAbductor
5B 2D 00 00 00 00 00 00 // eShip_UFOSupply
57 2D 00 00 00 00 00 00 // eShip_UFOBattle
58 2D 00 00 00 00 00 00 // eShip_UFOEthereal
6F 29 00 00 00 00 00 00 // eMPT_SniperSquaddie  // +8  // stand-in for FIGHTER
5F 29 00 00 00 00 00 00 // eMPT_HeavySquaddie   // +8  // stand-in for RAIDER
73 29 00 00 00 00 00 00 // eMPT_SupportSquaddie // +8  // stand-in for HARVESTER
5B 29 00 00 00 00 00 00 // eMPT_AssaultSquaddie // +8  // stand-in for ASSAULT CARRIER
5A 29 00 00 00 00 00 00 // eMPT_AssaultColonel  // +8  // stand-in for TERROR SHIP
53 2D 00 00 00 00 00 00 // eShip_MAX

It is possible to resize the upk and insert additional names, but in very few cases will this matter, so I've chosen here to use some name entries from the multiplayer template enum.

After resizing and inserting the new code, UE Explorer decompiles the enum as:

enum EShipType
{
    eShip_None,
    eShip_Interceptor,
    eShip_Skyranger,
    eShip_Firestorm,
    eShip_UFOSmallScout,
    eShip_UFOLargeScout,
    eShip_UFOAbductor,
    eShip_UFOSupply,
    eShip_UFOBattle,
    eShip_UFOEthereal,
    eMPT_SniperSquaddie,
    eMPT_HeavySquaddie,
    eMPT_SupportSquaddie,
    eMPT_AssaultSquaddie,
    eMPT_AssaultColonel,
    eShip_MAX
};

The game still runs normally after the above change. Further, the "clamping" that we had been experiencing has been resolved.

Below is the full upk_mod file I used to modify the upk, for reference.

MODFILEVERSION=4
UPKFILE=XComGame.upk
GUID=5B 06 B8 18 67 22 12 44 85 9B A8 5B 9D 57 1D 4B // EW Patch 1
FUNCTION=EShipType@XGGameData
RESIZE=28

// increase number of EShip enums

[BEFORE_HEX]
[HEADER]
7C 2E 00 00 76 5F 00 00 00 00 00 00 7C 2E 00 00 
0B 00 00 00 // length
[/HEADER]
54 2D 00 00 00 00 00 00 // eShip_None
52 2D 00 00 00 00 00 00 // eShip_Interceptor
55 2D 00 00 00 00 00 00 // eShip_Skyranger
51 2D 00 00 00 00 00 00 // eShip_Firestorm
5A 2D 00 00 00 00 00 00 // eShip_UFOSmallScout
59 2D 00 00 00 00 00 00 // eShip_UFOLargeScout
56 2D 00 00 00 00 00 00 // eShip_UFOAbductor
5B 2D 00 00 00 00 00 00 // eShip_UFOSupply
57 2D 00 00 00 00 00 00 // eShip_UFOBattle
58 2D 00 00 00 00 00 00 // eShip_UFOEthereal
53 2D 00 00 00 00 00 00 // eShip_MAX

[/BEFORE_HEX]


[AFTER_HEX]
[HEADER]
7C 2E 00 00 76 5F 00 00 00 00 00 00 7C 2E 00 00 
10 00 00 00 // length
[/HEADER]
54 2D 00 00 00 00 00 00 // eShip_None
52 2D 00 00 00 00 00 00 // eShip_Interceptor
55 2D 00 00 00 00 00 00 // eShip_Skyranger
51 2D 00 00 00 00 00 00 // eShip_Firestorm
5A 2D 00 00 00 00 00 00 // eShip_UFOSmallScout
59 2D 00 00 00 00 00 00 // eShip_UFOLargeScout
56 2D 00 00 00 00 00 00 // eShip_UFOAbductor
5B 2D 00 00 00 00 00 00 // eShip_UFOSupply
57 2D 00 00 00 00 00 00 // eShip_UFOBattle
58 2D 00 00 00 00 00 00 // eShip_UFOEthereal
6F 29 00 00 00 00 00 00 // eMPT_SniperSquaddie  // +8  // stand-in for FIGHTER
5F 29 00 00 00 00 00 00 // eMPT_HeavySquaddie   // +8  // stand-in for RAIDER
73 29 00 00 00 00 00 00 // eMPT_SupportSquaddie // +8  // stand-in for HARVESTER
5B 29 00 00 00 00 00 00 // eMPT_AssaultSquaddie // +8  // stand-in for ASSAULT CARRIER
5A 29 00 00 00 00 00 00 // eMPT_AssaultColonel  // +8  // stand-in for TERROR SHIP
53 2D 00 00 00 00 00 00 // eShip_MAX
[/AFTER_HEX]

dubiousintent · February 17, 2014

Created wiki article 'Enum Expansion - XCOM:EU 2012' from this.

-Dubious-

Amineri · February 17, 2014

Created wiki article 'Enum Expansion - XCOM:EU 2012' from this.

-Dubious-

Thanks :). I've looked it over and made a couple of minor edits for clarity, otherwise it looks good.

Amineri · February 17, 2014

It's also possible to use the same name multiple times within the same enum. The following vanilla XGGameData.ENumPlayers enum does just this:

enum ENumPlayers
{
    ENumPlayers,
    ENumPlayers,
    ENumPlayers,
    ENumPlayers,
    ENumPlayers_MAX
};

using hex:

80 AE 00 00 76 5F 00 00 00 00 00 00 BF 61 00 00 
05 00 00 00 -- length
01 2A 00 00 01 00 00 00 -- ENumPlayers (1)
01 2A 00 00 02 00 00 00 -- ENumPlayers (2)
01 2A 00 00 03 00 00 00 -- ENumPlayers (3)
01 2A 00 00 04 00 00 00 -- ENumPlayers (4)
02 2A 00 00 00 00 00 00 -- ENumPlayers_MAX

Note that the upper 4 bytes of the name reference now contain non-zero data, indicating that the name index isn't a 8 byte long, but instead a composite of 2 4byte ints.

In this case the enum values use the same name as the enumeration itself -- ENumPlayers.

Edited February 17, 2014 by Amineri

wghost81 · February 18, 2014

By UE standards it actually should look like this:

01 2A 00 00 01 00 00 00 -- ENumPlayers_0
01 2A 00 00 02 00 00 00 -- ENumPlayers_1
01 2A 00 00 03 00 00 00 -- ENumPlayers_2
01 2A 00 00 04 00 00 00 -- ENumPlayers_3

NameIndex is still 8 bytes long, as it is always 8 bytes long and consists of NameTableIdx and Numeric. Edited February 18, 2014 by wghost81

UPK file format

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Recently Browsing 0 members