UPK Utils

[wghost81]

Here are source and test files for my UPKUtils project:
https://github.com/wghost/UPKUtils

UPKUtils is a set of simple console applications for analyzing and patching uncompressed UPK files. Main "library" consists of two files: UPKUtils.h and UPKUtils.cpp. Other files are using this library to perform actual operations on UPKs:

• ExtractNameLists — extract all header info (package info and name/object lists) and echo it into console. Use redirection to put it into some text file.
• FindObjectEntry — find object entry by object name and extract its data into binary file (optional).
• MoveExpandFunction — move and/or expand function by its name.
• PatchUPK — patch UPK file using modfile.
• RepairFunction — discontinued, waiting for Amineri to finish her work. :smile:

Helper library to parse modfiles: ModParser.h, ModParser.cpp.

 

UPKUtils is C++11 Code::Blocks project. I use latest MinGW to build it.

 

UPKUtils compiled and ready to use with some examples on nexus:

http://www.nexusmods.com/xcom/mods/448

 

I will explain more about modfile format in following post.

[wghost81]

Modfile format

 

Modfile is a simple human-readable txt file which uses format similar to ToolBoks custom mod. PatchUPK is fully compatible (at least, I hope :smile: ) with ToolBoks custom mods: it can read and implement all the changes,

 

Modfile consists of keys (NAME=VALUE), sections ([sECTION_NAME]) and comments. Comments are begin with { and end with }. You may place comments anywhere, they also can be multiline. Right now PatchUPK is case-sensitive, so be careful! I'm not an expert on text parsing, so I made it as I could, sorry. :sad:

 

List of supported keys and their allowed values:

• MOD_NAME, AUTHOR and DESCRIPTION are self-explanatory. :smile: Value is simple text. Can be multiline.
• UPK_FILE - name of UPK file to patch.
• OFFSET - dec or hex (0x) value.
• FUNCTION - full function name to patch.
• REL_OFFSET - relative offset, dec or hex (0x) value. If absolute offset value is set through OFFSET or FUNCTION key, REL_OFFSET is added to that value, so you can patch parts of function code without a need to specify it's full offset.
• FUNCTION_FILE - file (with path) to read data from. Can be used with OFFSET or FUNCTION or by itself, if file name has Full.Function.Name.Function format.
• NAMELIST_NAME - change namelist name. Value is of OLDNAME:NEWNAME.
• EXPAND_FUNCTION - move and expand function by its name. Value is Full.Function.Name:NEWSIZE.

List of supported sections:

• [MODDED_HEX] - section which contains modded space-separated hex data (as in ToolBoks custom mod).
• [bEFORE_HEX] and [AFTER_HEX] - hex data for search-and-replace style patching,
• [/bEFORE_HEX] and [/AFTER_HEX] are permitted, by not used anyway, since parser uses different method to find data blocks.

Some examples:

 

Move and expand:

 

UPK_FILE=XComStrategyGame.upk
EXPAND_FUNCTION=XGStrategyAI.GetAltWeapon:300

Find and replace:

 

UPK_FILE=XComGame.upk

[BEFORE_HEX]
CC 04 00 00 18 03 00 00 0F 00 FF B2 00 00 25 07 C9 04 96 00 FF B2 00 00 2C 04 16 07 51 00 9A 38 3A 35 7A B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 38 3A 24 10 16 04 0B 06 BB 04 07 BB 04 9A 38 3A 35 7A B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 38 3A 24 00 16 0F 35 7A B2 00 00 7C B2 00 00 00 01 1A 00 FF B2 00 00 01 A6 B2 00 00 24 10 07 BF 03 81 2D 01 88 B2 00 00 16 0F 00 FE B2 00 00 25 07 BC 03 96 00 FE B2 00 00 36 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 07 DA 01 9B 38 3A 35 3D 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 38 3A 24 00 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 7D 00 1B 54 0F 00 00 00 00 00 00 35 3D 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 35 3A 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 16 07 C4 02 9B 38 3A 35 3C 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 38 3A 24 00 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 7D 00 1B 54 0F 00 00 00 00 00 00 35 3C 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 35 39 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 16 07 AE 03 9B 38 3A 35 3B 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 38 3A 24 00 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 7D 00 1B 54 0F 00 00 00 00 00 00 35 3B 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 35 38 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 16 A5 00 FE B2 00 00 16 06 C5 00 06 B8 04 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 2A 00 1B 54 0F 00 00 00 00 00 00 38 3D 35 6A B2 00 00 71 B2 00 00 00 00 01 82 B2 00 00 4A 16 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 2A 00 1B 54 0F 00 00 00 00 00 00 38 3D 35 69 B2 00 00 71 B2 00 00 00 00 01 82 B2 00 00 4A 16 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 2A 00 1B 54 0F 00 00 00 00 00 00 38 3D 35 66 B2 00 00 71 B2 00 00 00 00 01 82 B2 00 00 4A 16 16 06 C9 04 A3 00 FF B2 00 00 16 06 0B 00 04
[/BEFORE_HEX]

[AFTER_HEX]
A8 04 00 00 18 03 00 00 0F 00 FF B2 00 00 25 07 89 04 96 00 FF B2 00 00 2C 04 16 07 51 00 9A 38 3A 35 7A B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 38 3A 24 10 16 04 0B 06 7B 04 07 7B 04 9A 38 3A 35 7A B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 38 3A 24 00 16 0F 35 7A B2 00 00 7C B2 00 00 00 01 1A 00 FF B2 00 00 01 A6 B2 00 00 24 10 07 7F 03 81 2D 01 88 B2 00 00 16 0F 00 FE B2 00 00 25 07 7C 03 96 00 FE B2 00 00 36 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 0F 01 98 B2 00 00 38 3A 35 3D 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 07 84 01 97 01 98 B2 00 00 25 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 7D 00 1B 54 0F 00 00 00 00 00 00 38 3D 01 98 B2 00 00 25 16 16 0F 01 98 B2 00 00 38 3A 35 3C 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 07 79 02 97 01 98 B2 00 00 25 16 0F 00 57 B3 00 00 25 07 79 02 98 00 57 B3 00 00 35 39 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 7D 00 1B 54 0F 00 00 00 00 00 00 38 3D 01 98 B2 00 00 24 00 16 16 A5 00 57 B3 00 00 16 06 E3 01 0F 01 98 B2 00 00 38 3A 35 3B 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 07 6E 03 97 01 98 B2 00 00 25 16 0F 00 57 B3 00 00 25 07 6E 03 98 00 57 B3 00 00 35 38 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 7D 00 1B 54 0F 00 00 00 00 00 00 38 3D 01 98 B2 00 00 24 00 16 16 A5 00 57 B3 00 00 16 06 D8 02 A5 00 FE B2 00 00 16 06 C5 00 06 78 04 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 2A 00 1B 54 0F 00 00 00 00 00 00 38 3D 35 6A B2 00 00 71 B2 00 00 00 00 01 82 B2 00 00 4A 16 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 2A 00 1B 54 0F 00 00 00 00 00 00 38 3D 35 69 B2 00 00 71 B2 00 00 00 00 01 82 B2 00 00 4A 16 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 2A 00 1B 54 0F 00 00 00 00 00 00 38 3D 35 66 B2 00 00 71 B2 00 00 00 00 01 82 B2 00 00 4A 16 16 06 89 04 A3 00 FF B2 00 00 16 06 0B 00 04 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B
[/AFTER_HEX]

Rename function:

 

UPK_FILE=XComStrategyGame.upk
NAMELIST_NAME=GetAltWeapon:#GetPodProgs

Patch part of the function by relative offset:

 

UPK_FILE=XComGame.upk
FUNCTION=XGBattleDesc.InitAlienLoadoutInfos     
REL_OFFSET=0x28
[AFTER_HEX]
A8 04 00 00 18 03 00 00 0F 00 FF B2 00 00 25 07 89 04 96 00 FF B2 00 00 2C 04 16 07 51 00 9A 38 3A 35 7A B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 38 3A 24 10 16 04 0B 06 7B 04 07 7B 04 9A 38 3A 35 7A B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 38 3A 24 00 16 0F 35 7A B2 00 00 7C B2 00 00 00 01 1A 00 FF B2 00 00 01 A6 B2 00 00 24 10 07 7F 03 81 2D 01 88 B2 00 00 16 0F 00 FE B2 00 00 25 07 7C 03 96 00 FE B2 00 00 36 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 0F 01 98 B2 00 00 38 3A 35 3D 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 07 84 01 97 01 98 B2 00 00 25 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 7D 00 1B 54 0F 00 00 00 00 00 00 38 3D 01 98 B2 00 00 25 16 16 0F 01 98 B2 00 00 38 3A 35 3C 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 07 79 02 97 01 98 B2 00 00 25 16 0F 00 57 B3 00 00 25 07 79 02 98 00 57 B3 00 00 35 39 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 7D 00 1B 54 0F 00 00 00 00 00 00 38 3D 01 98 B2 00 00 24 00 16 16 A5 00 57 B3 00 00 16 06 E3 01 0F 01 98 B2 00 00 38 3A 35 3B 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 07 6E 03 97 01 98 B2 00 00 25 16 0F 00 57 B3 00 00 25 07 6E 03 98 00 57 B3 00 00 35 38 00 00 00 3F 00 00 00 00 00 10 00 FE B2 00 00 35 42 00 00 00 43 00 00 00 00 00 01 95 B2 00 00 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 7D 00 1B 54 0F 00 00 00 00 00 00 38 3D 01 98 B2 00 00 24 00 16 16 A5 00 57 B3 00 00 16 06 D8 02 A5 00 FE B2 00 00 16 06 C5 00 06 78 04 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 2A 00 1B 54 0F 00 00 00 00 00 00 38 3D 35 6A B2 00 00 71 B2 00 00 00 00 01 82 B2 00 00 4A 16 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 2A 00 1B 54 0F 00 00 00 00 00 00 38 3D 35 69 B2 00 00 71 B2 00 00 00 00 01 82 B2 00 00 4A 16 16 55 35 7B B2 00 00 7C B2 00 00 00 00 1A 00 FF B2 00 00 01 A6 B2 00 00 2A 00 1B 54 0F 00 00 00 00 00 00 38 3D 35 66 B2 00 00 71 B2 00 00 00 00 01 82 B2 00 00 4A 16 16 06 89 04 A3 00 FF B2 00 00 16 06 0B 00 04 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 53
[/AFTER_HEX]

There is a fully functional mod example in nexus mod file archive (Larger Pods), it produces exact same version as the one released for patch 1. Should be fully compatible with unpatched EW.

 

PatchUPK does not support unicode (not in text, nor in paths)! As I said, I'm no expert on parsing.

 

Command line format

PatchUPK modfile.txt [PATH_TO_UPKS]

 

PATH_TO_UPKS is optional, if omitted upks are searched in current working directory.

Edited December 8, 2013 by wghost81

[Nolanoth]

And um... how do you exactly install/use this tool? Any step by step instructions?

[wghost81]

It does not need any installation. It's a set of console programs with command line arguments. If you run a program without any, you will receive "help" line like this:

ExtractNameLists
Usage: ExtractNameLists UnpackedResourceFile.upk

So, in this case you need to type ExtractNameLists Path\to\XComGame.upk, for example, and see a wall of text on the screen. :smile: You may redirect program output with >

ExtractNameLists Path\to\XComGame.upk > list.txt

Other programs work the same way. PatchUPK is fully functional upk patcher and can be used to install/distribute mods. Although, right now I'm working on GUI wrapper for PatchUPK to make it more user-friendly.

[dubiousintent]

Okay, they are not 'installed' in the traditional sense. But where should they be placed in the steam tree so they get recognized as valid console command additions? They have to be placed somewhere.

 

Added to the Modding Tools wiki page under 'Unreal Game Tools'.

[wghost81]

dubiousintent, thanks!

 

You may put it anywhere, just give the correct path to upk file in the command line.

 

I usually copy upk files from game folder to different working directory and then start analysing/patching. Utils are in the same dir too. In this case you don't need to bother with paths and may use upk copies directly, without fear of breaking something during experiments.

 

BTW, I'm a Linux user, actually, so I do like console. :smile: I use FAR manager under windows, it has "old-school" look and feel, but is powerful and easy to use with console apps.

Edited December 10, 2013 by wghost81

[dubiousintent]

Oh: then we need to clarify if you are referring to the 'XCOM Console' or the 'Linux Console' / 'Windows command prompt'? Originally I thought you meant these were XCOM Console additions, but now I'm thinking you mean they are standalone command line tools. (Confused minds want to know! :blush: )

 

-Dubious-

[wghost81]

Oh, my bad! Of course, they are windows console applications, not XCOM console additions.

 

BTW, as those utils are written in plain c++, one could try and build them under linux or mac. There is no linux XCOM, but there is mac XCOM, afaik.

[dubiousintent]

Familiar terminology can be a slippery thing!

 

I've added the note about the C++ source being available to the Modding Tools entry. Thanks for pointing that out.

 

-Dubious-

[wghost81]

Oops... found a stupid mistake in the code which could lead to PatchUPK crash. Will update programs soon, but right now you should avoid any spaces before/after key and section name! Use KEY=VALUE and [sECTION_NAME] with immediate end of line to avoid any troubles!

 

Sorry for this. :sad:

 

UPD: updated both github sources and nexus files binaries to v.0.9.1

Edited December 10, 2013 by wghost81