UPK file format

[Drakous79]

I don't understand how to apply those flags - where it starts. From test in UE Explorer, I got this result:

native static final function bool

01 24 02 00

 

0x00000001 Final

0x00000400 Native

0x00002000 Static

 

Are only three 00 needed? Like 01 24 02.

Edited December 19, 2013 by Drakous79

[EliotVU]

I don't understand how to apply those flags - where it starts. From test in UE Explorer, I got this result:

 

native static final function bool

01 24 02 00

 

0x00000001 Final

0x00000400 Native

0x00002000 Static

 

Are only three 00 needed? Like 01 24 02.

 

Here's an article explaining this: http://www.cplusplus.com/forum/general/1590/

 

in UE Explorer I call them Flags, but in Computer Science they're usually referred as Bitmasks of bit flags.

Edited December 19, 2013 by EliotVU

[Amineri]

Really good find, wghost :D

 

That likely explains why my initial attempts to convert a native function into unrealscript didn't work.

 

One general issue I've been pondering is what the scope of this change is -- It certainly would affect all unreal calls to the native function, but I'm suspecting that it might not affect calls by the native code to the native function (e.g. one native function calling another).

[wghost81]

One general issue I've been pondering is what the scope of this change is -- It certainly would affect all unreal calls to the native function, but I'm suspecting that it might not affect calls by the native code to the native function (e.g. one native function calling another).

Yes, it most certainly wouldn't affect any C++ function calls.

[wghost81]

Small update on object table entry format.

 

I used Gildor's upk extractor to extract objects from map packages and found a large groups of objects with same name but different "count" added to that name. Like this:

FX_Alien_Destruction_Plasma.P_Power_Core_Destroyed.ParticleModuleColorScaleOverLife_16.DistributionFloatParticleParameter_0

"_16" and "_0" are object counts. Seems, that unknown field in object list entry following NameListIdx field determines this number. I called it NameCount. If NameCount == 0, name is unique and nothing happens. But if NameCount > 0, name is non-unique and string "_N" is added to full object name, where N = NameCount - 1 (wiki article updated).

 

Both UE Explorer and Gildor's upk extractor subtract 1 from NameCount, but UE Explorer does not add "_0" to names with NameCount == 1. Upk extractor does, so does my utility.

 

And a word on full object name format. Following UE Explorer and Gildor's utilities notation, I use Owner.Owner.Name structure for full object names. For dumping binary object data I use Owner.Owner.Name.Type as file name.

 

In new version of ExtractNameLists I will use Gildor's notation for object type and full name:

Type'Owner.Owner.Name'

If type points to null objects, it means that this object is a class, so "Class" is added to Type string:

0x104 (260): StrProperty'Checkpoint.GetBaseLevelName.ReturnValue'
0x105 (261): Function'Checkpoint.GetBaseLevelName'
0x106 (262): Class'Checkpoint'
0x107 (263): Checkpoint'Default__Checkpoint'
0x108 (264): Class'Checkpoint_StrategyTransport'
0x109 (265): Checkpoint_StrategyTransport'Default__Checkpoint_StrategyTransport'
0x10a (266): Class'Checkpoint_TacticalGame'

For Import Table objects scope resolution operator is used to indicate object's package:

0x8 (8): Engine::AnimNotify_PlayParticleEffect'Engine.Default__AnimNotify_PlayParticleEffect'
0x9 (9): Core::ArrayProperty'Core.Object.InterpCurveFloat.Points'

Edited December 21, 2013 by wghost81

[Amineri]

One of the things I've figured out how to do in the last month is change the type of a variable or parameter, but I haven't yet worked out a consistent way to describe such a change in a manner to make the change as patch-resistant as possible.

 

I'll describe this using a real example of something I want to change. My plan is to take over the XGUnit.DebugAnims function to use as a helper for creating individualized alien upgrades. For those of you that played Long War this is how the alien leaders were added. Here is the prototype for DebugAnims:

simulated function DebugAnims(Canvas kCanvas, XComTacticalCheatManager kCheatManager)

The call to DebugAnims is being made from XComAlienPodManager.OvermindSpawn. For the EU version of the mod the relevant code section looked like:

if (iNum == 0) // only apply level to leader
{
	kAlien.m_iSmokeGrenades = 0x7 & int(kSpawn.kPod.eMainAltWeapon); // store the main alien's level in its unit
}
kAlien.DebugAnims(none, none); // call the helper function to "level up" potentially any unit

I needed to pass the an integer to DebugAnims but DebugAnims only parameters are classes. So I hacked a workaround of putting the value in a class variable and reading that value in DebugAnims -- in this case m_iSmokeGrenades.

 

However in EW this parameter is used by enemies -- EXALT.

 

So, what I'd like to do is change DebugAnim's first parameter type from Canvas kCanvas to int kCanvas.

 

Making this change requires changing the object table entry for kCanvas, however. The full name reference for the parameter is XGUnit.DebugAnims.kCanvas (in UPK modder to keep this distinct from class instances I write it kCanvas@DebugAnims@XGUnit).

 

The object table entry for kCanvas is :

76 FE FF FF 00 00 00 00 D6 C8 00 00 75 47 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 04 00 07 00 
2C 00 00 00 EA 44 A4 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00

There are two things to be changed to alter this from a class variable to an int variable :

1. The first int 76 FE FF FF = -394
   • This is an import table reference, and for this upk resolves to "ObjectProperty"
   • For different upks the value that matches to "ObjectProperty" will be, in general, different
   • This needs to be changed to 7A FE FF FF = -390, which is the import table reference for IntProperty
2. Class variable objects are 4 bytes larger that primitive variable objects -- the last 4 bytes holds the reference to the specific object type
   • To change the size of the referenced object the 2C 00 00 00 = 44 bytes needs to be changed to 28 00 00 00 = 40 bytes

 

For comparision here is the local variable iMirror's object table entry (from the same XGUnit.DebugAnims function):

7A FE FF FF 00 00 00 00 D6 C8 00 00 A8 40 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 04 00 07 00 
28 00 00 00 F6 43 A4 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00

The referenced size is 28 00 00 00 and the type is 7A FE FF FF.

 

Note that the types are not guaranteed to be consistent. In XComStrategyGame.upk the import reference for IntProperty is E6 FE FF FF = -282. The difference is because of the different number of import table objects. This means that using hex values directly to alter the type of a variable can be broken when a patch/update is released.

 

----------------

 

So my goal here is to stimulate some discussion about a common format for applying such changes to upks that is less likely to break when patches are released.

 

Finding the Object Table Entry would involve looking up the entry index based on name, in this case XGUnit.DebugAnims.kCanvas. This will give the index into the Object Table.

 

To change the type from ObjectProperty to IntProperty, the safest way is to lookup each of these names' import table index. Technically it's not required to lookup ObjectProperty, but it can be done as a form of error check. The first integer in the object table entry is always the type, so blind writing the index of IntProperty should in theory be okay.

 

Since changes to the original type of variable would mean more substantial changes, it is safe to change the size of the referenced object from 44 to 40. The object itself does not have to be changed, because the specific object reference is the last word in the object, so shortening the size effectively truncates it.

 

 

 

So, the key pieces of info need to make the change are:

• 'XGUnit.DebugAnims.kCanvas' -- The named reference of the object, used to retrieve its object table entry index
• 'ObjectProperty' -- the original type (optional-used for error checking or reverting) used to retrieve the original type from the import table
• 'IntProperty' -- the new type, used to retrieve the new type from the import table : : written to word 0
• 44 -- old size (optional-used for error checking or reverting)
• 40 -- new size : : written to integer word 8 (counting from 0)

 

My initial proposal is that the format look something like :

MODFILEVERSION=4
UPKFILE=XComGame.upk 
GUID=None
OBJECT=kCanvas@DebugAnims@XGUnit

[TYPECHANGE]
[BEFORE]
OBJECT_TYPE=ObjectProperty
SIZE=2A // hexadecimal
[/BEFORE

[AFTER]
OBJECT_TYPE=IntProperty
SIZE=28 // hexadecimal
[/AFTER]
[/TYPECHANGE]

Definitely open to ideas on this.

 

 

I made this somewhat general because there are other mods that can be made to variables... in particular changing the size of a static array comes to mind.

[wghost81]

Full Import Entry name for ObjectProperty looks like this:

Core::Class'Core.ObjectProperty'

i.e.

Package::Type'Owner.Name'

This name is unique and thus can be located in Import Table. So is IntProperty:

Core::Class'Core.IntProperty'

Export Table Entry format depends on object type, so we need to figure out entry/data format for each type and write the safe functions to deal with such king of changes. In PatchUPK-style :smile: script could look like this:

EXPORT_ENTRY = XGUnit.DebugAnims.kCanvas { find object entry in Export Table }
TYPE_CHANGE = ObjectProperty:IntProperty { change object type }

Or even shorter:

TYPE_CHANGE = XGUnit.DebugAnims.kCanvas:IntProperty { find object entry AND change object type }

BTW, here are some updates on Export Entry format. Copying/pasting from UPKUtils source without editing (don't have much time right now):

struct ObjectListEntry
{
    int32_t  ObjTypeRef;
    int32_t  ParentClassRef;
    int32_t  OwnerRef;
    uint32_t NameListIdx;
    uint32_t NameCount;
    uint32_t Field6;
    uint32_t ObjectFlagsH;         // ObjectFlags_H
    uint32_t ObjectFlagsL;         // ObjectFlags_L
    uint32_t ObjectFileSize;       // SerialSize
    uint32_t DataOffset;           // SerialOffset
    uint32_t Field11;              // ExportFlags
    uint32_t NumAdditionalFields;  // NetObjectCount
    uint32_t Field13;              // Guid_A (unused)
    uint32_t Field14;              // Guid_B (unused)
    uint32_t Field15;              // Guid_C (unused)
    uint32_t Field16;              // Guid_D (unused)
    uint32_t Field17;
};

Edited December 23, 2013 by wghost81

[Amineri]

Here is the .upk_mod file that my current testing version of UPKmodder uses to change object types:

MODFILEVERSION=4
UPKFILE=XComGame.upk 
GUID=UNKNOWN // XComGame_EW_patch1.upk
OBJECT=kCanvas@DebugAnims@XGUnit
ACTION=typechange

[BEFORE_HEX]
OBJECT_TYPE=Core:ObjectProperty@Core
SIZE=2C // hexadecimal
[/BEFORE_HEX]

[AFTER_HEX]
OBJECT_TYPE=Core:IntProperty@Core
SIZE=28 // hexadecimal
[/AFTER_HEX]

I've allowed to use the OBJECT= keyword instead of FUNCTION= (although under the hood there's not yet any explicit error checking).

 

I re-used the BEFORE_HEX and AFTER_HEX keywords just because I wanted to maintain backwards compatibility with modfile I've already written :p

 

I use a somewhat abbreviated form compared to the full form you use (and of course use specific-to-general order instead of general-to-specific). I haven't found it to be necessary to include the type in every searchable object name in order to keep them unique.

 

I explicitly leave the hooks open for the memory size change because ... well ... we're modders, so I didn't want to hide something that could be useful :)

 

Otherwise:

EXPORT_ENTRY = XGUnit.DebugAnims.kCanvas
  maps to
OBJECT=kCanvas@DebugAnims@XGUnit

and

TYPE_CHANGE = ObjectProperty:IntProperty

is broken down into:

ACTION=typechange

[BEFORE_HEX]
OBJECT_TYPE=Core:ObjectProperty@Core
[/BEFORE_HEX]

[AFTER_HEX]
OBJECT_TYPE=Core:IntProperty@Core
[/AFTER_HEX]

Your notations is definitely more compact than mine, for sure.

[wghost81]

Looks good, but we need to make a clear difference between object table entries and object data ("serial" in UE terminology). Furthermore, for some objects there also are Default Properties (which are objects themselves, but they are "connected" with respective objects).

 

So, I want to propose something like this:

NAME_ENTRY=Name or Index { search for Name Table entry }
EXPORT_ENTRY=Name or Index { search for Export Table entry }
IMPORT_ENTRY=Name or Index { search for Import Table entry }
OBJECT=Name or Index { search for Export Object serialized data }

PatchUPK already supports low-level and high-level changes: by telling the program to locate specific object you set current package file offset to that object start. And then you can do anything you want without any error checks. I will definitely keep this functionality for flexibility sake and add more "safe" keys to deal with upk patching.

[Amineri]

So a couple of new interesting issues have popped out...

 

First, there is some sort of hex code that I have been as yet unable to decipher.

 

I typically comes in at the end of a state's default function (not a subfunction within the state).

 

An easy example is XGAction_Overwatch.state_Executing. UE Explorer decompiles this as :

simulated state Executing
{
Begin:
    Sleep(1.750);
    CompleteAction();
    stop;        
}

Quite straightforward.

 

However the token view shows some additional hex at the end :

(000/001) [00 1E 00 00 E0 3F 16]
	NF(8/7) -> FC(5/5) -> EFP(1/1)
	Sleep(1.750)

(008/008) [1C 9C 8D 00 00 16]
	FF(10/6) -> EFP(1/1)
	CompleteAction()

(012/00E) [08]
	S(1/1)
	stop

(013/00F) [0C B2 09 00 00 00 00 00 00 00 00 00 00 76 5F 00 00 00 00 00 00 FF FF 00 00]
	LT(25/25)
	
(02C/028) [53]
	EOS(1/1)

This next-to-last hex line starts with 0x0C, which is a "Lable Table Token" and is then followed by 24 bytes.

 

Looking for possible interpretations, the B2 09 00 00 looks like it could be a namelist reference to "Begin", while 76 5F 00 00 looks like it could be a namelist reference to "None"

 

So in general it looks like the token parses as 0C <namelist reference> 00 00 00 00 <4 bytes> <namelist reference> <4 bytes>

 

I'm starting to think that for some reason the namelist indices are stored as long ints (which is why there's always the extra 4 00 "padding", although why Epic ever thought there'd be a need for > 4 billion names ... I don't know!