UPK Utils

[wghost81]

I'll put GUIDs on To-Do list along with auto-expand function feature. :smile:

 

BTW, if someone in Firaxis decide to delete one or two variables from the list and you just happen to use last object in the list inside the mod, reference will be invalid. So it is something worth checking.

[Amineri]

Oh definitely... EU XComGame has ~5000 fewer export table entries (~50,000 compared to ~55,000) so there's definitely some chance of it:)

 

The fun part of updating across expansion is that it's not just 5000 more entries. Some EU variables/functions were removed, and so no longer even map to anything. Trying to deal with all of those permutations when trying to build this new tool has been an interesting exercise ^.^

 

------------

 

@wghost81

 

I'm curious if you see the various tools that you are developing aimed more at making mod installation easier or mod development easier? Or is it more of a mix?

 

On the one hand you have the tool that allows for cross-version installation of simple mods (which seems more installation-oriented), but on the other you built the tool to allow resizing of functions in a upk (which distinctly appeals to making mod development easier). :)

[wghost81]

Amineri, I don't really know. :smile: I started to work on upk tools when I realized I need more information on what I'm working with and more convenient way to implement changes/distribute mods. If you remember, my first question here was on how I can distribute my work. :smile:

 

So, I guess, my tools are more of a mix. When you posted some info on upk format and initial ideas on automatic script "repair" for new patches, I immediately started analyzing upk files and searching for new info. That's how my first tool — ExtractNameLists — was written. When I realized I need a tool to quickly find object info and extract it's data. That was my second tool — FindObjectEntry. When I started to work on RepairFunction tool, but soon realized, I'll end up writing a decompiler. And since you were already working on this, I switched to PatchUPK, as I always wanted more flexible way to install mods.

[Bertilsson]

I guess I should explicitly state something that may not be obvious to everyone...

 

If I have a bunch of hex references in some mod code, those hex references will validly point to objects in any game version upk. It's just that there is only one game version in which they are interpreted correctly.

 

These references are not some sort of sparse hash -- they are a sequential index to an array. So reference 10 A0 00 00 can be interpreted as:

• PlayerIndex@InitAlienLoadoutInfos@XGBattleDesc -- XComGame.upk for EU patch 4 <== correct interpretation
• InitHumanLoadoutInfosFromProfileSettingsSaveData@XGBattleDesc -- XComGame.upk for EU patch 5
• kNullCover@InitFromPlayer@XGAIBehavior_FlyingUnit -- XComGame.upk for EW patch 0

It's even possible to match a hex reference value to the wrong upk.

 

For example : FE 2C 00 00

• XComGameReplicationInfo -- XComGame.upk for EU patch 4 <=== correct interpretation
• ReturnValue@GetLockerItem@XGFacility_Lockers -- XComStrategyGame.upk for EU patch 4

With multiple versions of the game running around its very easy to get these wrong, so I'm using the GUID as a way to fairly strictly manage the versions.

Is it possible to dumb this down a little so I can get my head around it?

I'm not understanding in which scenario this becomes an issue...

 

If you have named references then you would "simply" look-up the byte code reference?

If you have have modded byte code that needs to be translated to named reference, wouldn't it be _very_ well known exactly which upk file and changelist version the mod was made for?

 

Not trying to put anyone in a spot or anything, just trying to catch up and right now I'm not even understanding what the problem is :unsure:

[Amineri]

For a single mod that you made yourself, the situation is pretty straightforward. You know what version of the game the mod was for, so you directly indicate which upk to use when looking up hex references.

 

However, if you didn't create the mod, things may not be so clear. It all depends on whether/how the author specified the version. This is probably not a big deal most of the time.

 

Now we get to my case (which is probably rather rare, but heck ... I care about it :)) : I've created on the order of 20 or so different mods. Some of these mods are quite complex, involving the coordinated rewriting of 10 to 15 functions for a single mod. At this point having some tools to help me sort out game versions gets useful. The issue is just plain human error and not remembering for all 150 different functions which version each is updated to.

 

Then we get to something like Long War which has changes to over 300 different functions. Trying to update such a mod to a new version is definitely no trivial task. I was looking for a way to clearly and unambiguously know what game version a upk is for.

 

The current method is based on the version.txt changelist -- which is not an intrinsic part of the upk. Plus I was just reading on the forums that Firaxis released a "stealth" patch where they didn't update the changelist in version.txt.

 

The safest way is just to embed the GUID itself into the modfile as a part of the descriptor. I admit it's a bit of overkill, like using 1,048,576 bit encryption. It can tell different upks from each other in the same version, between versions, even upks from other games. But the only part I care about is that it can distinguish between one version and another. Unless Firaxis specifically overrides it and forces the GUID to a specific value, every time they compile a new version of the game those upks will have new GUIDs.

 

My current design for my modfile has a header like so :

MODFILEVERSION=3
UPKFILE=XComGame.upk 
GUID= 33 2E 29 6A A5 DD FC 40 B5 CC 57 A5 A7 AA 8C 41 // EU patch 4
FUNCTION=ApplyActionCost@XGAbilityTree

• MODFILEVERSION is for tracking my own changes to my modfile format (so I can do error checking and maintain backwards compatibility)
• UPKFILE and GUID are a bit redundant .. technically only GUID is needed for checking, but giving the filename helps improve readability. It also gives hints to the tool to help find the correct upkfile.
• FUNCTION is used to narrow down the search region that the hex change can be applied to in the function, speeding it up considerably. Reducing the scope of the search and replace also limits the chance of "false positive" matches if the hex happens to match some changed code in another region of a new game version's upk

Keep in mind that all of this is designed around "heavy" mods with a ton of change. I in no way am suggesting that this kind of overhead is necessary in all cases.

[Drakous79]

What about multiple GUIDs checking? Some mods can be applied to more versions, because nothing changed.

[wghost81]

What about multiple GUIDs checking? Some mods can be applied to more versions, because nothing changed.

Yes, this will be a list of GUIDs like:

GUID=GUID_1
GUID=GUID_2
GUID=GUID_3
...

But the best solution for simple mods will be not to put any GUID check at all. And, of course, don't change modified upk GUID.

[wghost81]

Seems PatchUPK modfile keys turned to whole new script language. :smile: I decided to rewrite the whole thing to make it more flexible. It will not affect end-user interface, but will make source code more readable and easily expandable. And since holidays are coming, I will be busy and won't be able to add new functionality (like GUIDs) soon. But I will return to this in next 2-3 weeks.

[Drakous79]

Enjoy holidays :)

Edited December 23, 2013 by Drakous79

[wghost81]

Thanks! :smile: But there still is an entire week of hard work before I could enjoy anything. :smile: