UPK file format

[Amineri]

Just to clarify a bit about the word "offset", since it is being a bit overloaded here (I frequently use it in other contexts as well).

 

The offsets wghost81 is refering to are file offsets, which are measured from the start of the upk file. There are all measured in file byte size.

 

However within a particular function's hex I often refer to absolute and relative jump offsets. Within the context of a function an absolute jump offset measures a jump's target position relative to the first byte following the function header. A relative jump offset measures a jump target's positions relative to the current positions. Both of these are measured in memory byte size.

 

As it turns out memory byte sizes aren't that hard to compute once you know where all of the references are, and how many. Every reference adds +4 memory bytes (only Import/Export table references, not virtual function references).

[dubiousintent]

Yes, indeed ''file'' and ''jump'' offsets are the distinctions I was trying to recall. I think that helps the article. Updated.

 

-Dubious-

[Bertilsson]

Just a little question out of curiosity...

 

Is it possible to move functions between classes with current understanding of the UPK file format?

 

Example: XGFacility_Barracks.GetNumSoldiersOfClass ---> XGBase.GetNumSoldiersOfClass

[Amineri]

I'm not 100% sure (and would love to be proven wrong) but it appears like it's going to require a lot of rewriting of the package file to move a function from one class to another.

 

It appears that all of the parameters+localvars for each function have to be "packed together" and that the function defines which elements it owns by defining the start/end points of this range.

 

The example function I looked at had reference values for its parameters+localvars of:

UIAnchoredMessageMgr.Message default value in UE Explorer Managed Properties : D8 18 00 00 

parameters:
_sMsg,                  : D7 18 00 00
_xLoc,                  : D6 18 00 00
_yLoc,                  : D5 18 00 00
_anchor,                : D4 18 00 00
_displayTime,           : D3 18 00 00
_sId,                   : D2 18 00 00
_iIcon,                 : D1 18 00 00
_eBroadcastToTeams      : D0 18 00 00

return value (skipped if not present):

locals:
kMsg;                   : CF 18 00 00
bDisplayMessage,        : CE 18 00 00
bBroadcastMessage;      : CD 18 00 00
ibDisplayMessage,       : CC 18 00 00
ibBroadcastMessage;     : CB 18 00 00
kBroadcastMessage;      : CA 18 00 00 (6346)

Similarly, all of the functions within a class appear to have their parameter+localvar ranges consecutively ordered, along with the class variables. For example :

UIAnchoredMessageMgr.idCounter : C2 18 00 00
UIAnchoredMessageMgr.m_MessageContainer : C1 18 00 00
UIAnchoredMessageMgr.m_arrMsgs : C0 18 00 00

Functions in UIAnchoredMessageMgr:
Init                    : C2 18 00 00 - C4 18 00 00
OnInit                  : C5 18 00 00 - C5 18 00 00
OnCommand               : C6 18 00 00 - C8 18 00 00 
Message                 : C9 18 00 00 - D7 18 00 00
CreateMessage           : D8 18 00 00 - DD 18 00 00
RemoveMessage           : DE 18 00 00 - DF 18 00 00
AS_RemoveMessageBox     : E0 18 00 00 - E1 18 00 00
Pause                   : E2 18 00 00 - E2 18 00 00 
Resume                  : E3 18 00 00 - E3 18 00 00
Hide                    : E4 18 00 00 - E4 18 00 00
RemoveMessageData       : E5 18 00 00 - E7 18 00 00
IsMessageShown          : E8 18 00 00 - EB 18 00 00
EnableDepthSorting      : EC 18 00 00 - ED 18 00 00 

I don't know if this sequential ordering is required, but it appears that is how the unreal compiler generates the current upks.

 

I suppose with a little bit of careful management it might be possible to reduce the number of parameters+localvars and create enough space in the object table to add another function.

 

It also might be possible to move a function from an adjacent class if there aren't any class variables.

[Bertilsson]

Thank you for the quick and detailed answer.

 

Was just curious and don't really see any need to move functions since the most obvious reason to use an helper functions is mitigated by the possibility to expand function sizes :)

[Amineri]

So as a minor curiosity I just found a relatively uncommon second operand token that utilizes the "virtual function" type of reference (that is, an index to the name list instead of Export or Import table).

 

It's the 0x43 token (Delegate token). Apparently it is used to make something akin to function pointers that can be passed as arguments to other functions.

 

I came across it in XGFacility_Labs.OnResearchComplete in the line :

PRES().UINarrative(xcomnarrativemoment'AlienBase', none, ResearchCinematicComplete)

It turns out that ResearchCinematicComplete is a Delegate function, which is represented in byte code via :

43 DC 22 00 00 00 00 00 00 00 00 00 00 

The DC 22 00 00 reference has to be resolved as a name list reference, not as an Import/Export list reference.

 

Good times :)

[Amineri]

I'm wondering if anyone has found any header information anywhere in the upk that identifies where the Actionscript GFX files are?

 

A brief history of what has been found so far (more info can be accessed on the wiki : http://wiki.tesnexus.com/index.php/Command1.upk) :

 

In Enemy Unknown most strategy game interfaces were in the Command1.upk and most tactical game interfaces were embedded into the maps themselves (meaning that alterations had to be applied to every mapfile ~_~).

 

In Enemy Within most actionscript appears to have migrated into a collection of UICollection_*_SF.upk files (e.g. UICollection_Strategy_SF.upk for strategy game actionscript).

 

I know that the actionscript files can be found by searching for the text key "GFX" which heads each embedded file. (however not every instance of "GFX" in a file corresponds to an embedded actionscript file).

 

Following "GFX" is a header which conforms to the actionscript header format (and extraction and renaming of "GFX" -> "FWS" allows the extracted portion to be read with an SWF parser) which looks like :

47 46 58 -- ASCII chars 'G' 'F' 'X'
09 -- single byte version ID
D0 7D 01 00 -- size of embedded file (in bytes)
80 00 03 20 00 00 01 C2 00 00 1E 01 00 1C FA 6E 03 13 00 00 00 0D 00 00 -- some actionscript stuff
12 -- size of name string
41 6E 63 68 6F 72 65 64 4D 65 73 73 61 67 65 4D 67 72 -- name string "AnchoredMessageMgr"

and conforms to the SWF header spec. (This is the first embedded actionscript file in UICollection_Strategy_SF.upk).

 

The next GFX header starts with (as text):

GFX.üf..€.. ...Â......ún.........AugmentSoldier

However I haven't been able to discover any header information or tables in the upk that point to where the embedded flash files are.

 

Also some GFX chunks in UICollection_Strategy appear to be version 9 and some appear to be version 8.

 

The reason I'm wondering is that I've just finished updating the actionscript portion of "Expanded Perk Tree" mod for EW but don't have any way to localize the search and replace below the file level. All of these changes are made to the embedded flash file at :

GFX.$Œ..€.. ...Â......ún.........SoldierPromotion

I know that a few have built tools to automated extract the embedded flash so I'm hoping that there's a quick way to find the various upk sections.

[XMarksTheSpot]

Does a UPK header's object lists reference only UnrealScript-related items or do they also reference embedded resources, i.e. models, textures and the like? If it's the latter I'd hazard a guess and say that flash files are also a type of embedded resource and should show up somewhere, maybe you could try looking up an export/import entry by its file offset (or the closest offset you can find near where there's a GFX in the file)?

[Amineri]

I checked in UICollection_Strategy_SF.upk and the named object in the objectlist gfxSoldierPromotion has an Export Table entry of :

FF FF FF FF 00 00 00 00 00 00 00 00 17 01 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 04 00 07 00 
0C 00 00 00 23 75 01 00 01 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
05 00 00 20

so is at file position 0x17523 and only has a size of 0xC = 12 bytes.

 

The object pointed to is :

FF FF FF FF B0 01 00 00 00 00 00 00

which is part of a big sequence of identical objects.

 

The actual embedded GFX files begin appearing after that sequence.

 

The SoldierPromotion "GFX" tag is at offset 0x76EEF2, significantly further into the upk.

 

The SoldierPromotion actionscript appears to be #48 (when I counted by searching for "GFX" and not counting those in the middle of a GFX file). The only correspondence I can find is that the "FF FF FF FF B0 01 00 00 00 00 00 00" pointed to by gfxSoldierPromotion is also #48 in the list of identical objects.

Edited December 18, 2013 by Amineri

[Amineri]

So apparently I was looking at the wrong object... UE Explorer hides 'non-supported types' from displaying in the Objects pane.

 

Here's the "SoldierPromotion" object info as an example, and how the object corresponds to the SWF embedded file.

 

The correct object (the actual SWF actionscript) is named "SoldierPromotion" and has Export Table entry :

FC FF FF FF 00 00 00 00 31 00 00 00 5A 02 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 04 00 0F 00 
6D 8E 01 00 A9 EC 76 00 01 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00

It points to 0x76ECA9 and has a quite substantial size -- 0x18E6D = ~100kB, which is about right.

 

Note that the object's position is the above 0x76ECA9 while the GFX tag is at 0x76EEF2, a couple hundred bytes into the object. Also the object size is 0x18E6D while the GFX/SWF embedded file size is 0x18C24. So the SoldierPromotion object definitely contains some header/wrapper info around the GFX file.

 

The SoldierPromotion upk object also contains some links to bitmap resources, for example:

TGA.................L...............gfxXComIcons.XComIcons.)...gfxStrategyComponents.StrategyComponents

This appears inside the SoldierPromotion upk object but before the GFX portion.

 

 

For my purposes I think the UPKmodder tool I'm working on will search the correct upkfile/portion of file if I include the header:

UPKFILE=UICollection_Strategy_SF.upk 
GUID=7B 52 25 96 F3 8E 7C 48 8F 11 3B 40 43 3F BD 3F  // UICollection_Strategy_SF_EW_patch1.upk
FUNCTION=SoldierPromotion@gfxSoldierPromotion

And yes, the UICollection*.upk's have GUIDs just like every other upk :)