UPK Utils

[wghost81]

Embedding exe upk names replacement in PatcherGUI idea turned to be bad. XComGame.exe is about 27 MB and I'm not an expert in fast search/replace algorithms. I tried to use std C++ search algorithm, but it was still too slow. And replacing upk names by offset will make patcher version-dependent.

 

I did a quick search and found an easy-to-use and fast command line utility FileBinReplace.exe. Syntax example:

FileBinReplace.exe "XComGame.exe" xcomgame.upk ycomgame.upk

Since this needs to be done only once (for each upk included in exe), it might be easier to write standalone batch file for this purpose.

[wghost81]

Updated git/release with new features: BYTE, INTEGER and FLOAT keys, multiple uninstall scripts to prevent accidental overwrites by subsequent installation of the same mod.

 

Added new install scripts for Larger Pods and Camera Zoom Level mods. Camera mod is EU/EW compatible.

Edited December 12, 2013 by wghost81

[dubiousintent]

Have you considered support for (optionally) verifying UPKs by their GUIDs, as suggested by Amineri elsewhere? Of course you would need to add a mechanism to parse and report them from the UPK files in the first place. But I'm also wondering if we can figure out how to generate our own? Then it would be possible for each mod to a UPK to have it's own GUID and thus be able to know if it was parsing it's own version or an vanilla version or some other modded version. That (what they use as a salt to the hash that is the GUID) has probably been figured out elsewhere in the Unreal community, one would think.

 

-Dubious-

[wghost81]

I can extract and report GUID from upk files, but the question is, how can we generate our own GUIDs? I found this info:

 

 

Every package has a GUID associated with it, which stands for Globally Unique Identifier. Every package has a GUID generated for it when it is saved. This GUID can then serve to identify different version of the same package, or to differentiate between packages. The GUID takes into account the time and other information in generating a 128-bit ID, so you can be sure that no two GUID's will ever be alike through pure chance. There are 2128 = 2.4 x 1038 possible GUIDs. I'm sure that's enough to justify not having to worry about the problem. :smile:

 

After Epic went to the trouble to ensure that GUIDs could never repeat through chance, they created a way for you to do this on purpose. In the process of conforming a package, the new package is given the same GUID as the original package, among other interestng details. They are deemed to be alike for all intents and purposes. We'll see why this in a later section.

 

As I understand, GUID is used in multiplayer games only, so theoretically we can utilize them for our own purposes.

 

I can add something like

UPK_GUID=upk-guid-here
UPK_GUID=another-upk-guid-here
UPK_GUID=...

to PatchUPK functionality. PatchUPK will extract GIUD from upk before patching and compare it to GUID list. If no matches are found, it will report incompatibility error. Edited December 13, 2013 by wghost81

[dubiousintent]

Based upon this Wikipedia article, sounds like they are using a sequential algorithm. In which case you just need to identify the version of the GUID generator technique used, which in turn determines which bits contain the time. But most likely it's using a "COMB" ("combined guid/timestamp") technique, which replaces the last 6 bytes of Data4 in a random (version 4) GUID with the least-significant 6 bytes of the current system date/time.

 

And then there is always the 'endian question'.

A GUID's representation can be little endian or big endian, so all APIs need to ensure that the correct data structure is used.

Your source comments might point out where someone needs to pay attention when cross-compiling for other platforms.

 

-Dubious-

[Amineri]

My understanding is that the GUID is generated by the compiler at compile time. It's basically used to ensure that two versions of the game are exactly identical.

 

It enables an easy synchronization for multiplayer to verify that the two players are using identical game versions. Since games like Unreal Tournament allow for modding of individual files, this integrity check extends to the upk-to-upk level and isn't just at the game-to-game level.

 

We aren't modding multiplayer here (or at least I'm not), but these GUIDs can serve as a convenient way to tell different version from each other. Even with EU patch 4 to patch 5, with the XComStrategyGame.upk which did not change at all (to the point where the patch 4 version can be used in place of the patch 5 version), the GUID changed.

 

I have not verified (nor can I by myself) whether different localized versions may have different GUIDs. I highly doubt it since it would require a separate compile, which would make QA a nightmare. It should be that localization is done purely through the localization files. But it's possible ...

[wghost81]

I don't think changing GUID for every small mod will be a good idea, since most of the existing mods are compatible. But for big projects like Long War changing GUID is reasonable precaution. On the other hand, unistall feature allows to safely install and uninstall mods in case it turns to be incompatible. Edited December 13, 2013 by wghost81

[dubiousintent]

Hey, as long as the support for checking GUIDs is there, the question of whether or not to change one for a mod becomes an option for the mod author to decide. If the GUIDs listed in a mod don't match, it's an incompatibility. If no GUID is listed, it doesn't matter to the mod. The issue of how to create one can be put off until it becomes pertinent. (Just raising it now for consideration. Later on, it may already be solved, or we will all have moved on and it becomes someone else's problem.)

 

-Dubious-

[Amineri]

Fair enough :)

 

Since the new tool X and I are working on is designed around making it possible to create large mods (in the presence of Firaxis putting out updates), the GUID is a core part of being able to manage updating hex references from one version to another.

 

Our UPK parser already automatically extracts the GUID whenever the upk header is read. The reference updater assistant prototype already automatically updates the GUID to the target upk version when updating to a new version occurs.

 

My reasoning (I can't vouch for XMTS's ...) to leave the GUID as a text field is to allow the modder access to control/inspect as required. It's stored as hex like any other hex in the file, a la :

GUID= 33 2E 29 6A A5 DD FC 40 B5 CC 57 A5 A7 AA 8C 41 // XComGame_EU_patch4.upk

Totally agree that for small mods that only change constant values at fixed offsets (relative to function start position) it is totally not required.

 

For mods with lots of total function rewrites though it's going to be key to managing different game versions.

[Amineri]

I guess I should explicitly state something that may not be obvious to everyone...

 

If I have a bunch of hex references in some mod code, those hex references will validly point to objects in any game version upk. It's just that there is only one game version in which they are interpreted correctly.

 

These references are not some sort of sparse hash -- they are a sequential index to an array. So reference 10 A0 00 00 can be interpreted as:

• PlayerIndex@InitAlienLoadoutInfos@XGBattleDesc -- XComGame.upk for EU patch 4 <== correct interpretation
• InitHumanLoadoutInfosFromProfileSettingsSaveData@XGBattleDesc -- XComGame.upk for EU patch 5
• kNullCover@InitFromPlayer@XGAIBehavior_FlyingUnit -- XComGame.upk for EW patch 0

It's even possible to match a hex reference value to the wrong upk.

 

For example : FE 2C 00 00

• XComGameReplicationInfo -- XComGame.upk for EU patch 4 <=== correct interpretation
• ReturnValue@GetLockerItem@XGFacility_Lockers -- XComStrategyGame.upk for EU patch 4

With multiple versions of the game running around its very easy to get these wrong, so I'm using the GUID as a way to fairly strictly manage the versions.