R&D - New Modding Tools

[Amineri]

I have inefficient upk-to-upk matching code right now (it takes about 15 seconds to match all 50,000 entries between XComGame.upk patch 4 and 5), but it is at least working.

 

It appears that patch 5 removed 4 variables (all of them local variables name fDot) related to the flanking calculations, and added 21 new variables relating to flanking calculations and alien pod movement. These appear targeted at fixing the two most prominent bugs (flanking and teleporting).

 

The variables removed in XComGame.upk patch 5 were:

 

 

Objectlist[12705]: A1 31 00 00  : fDot.IsFlankingCoverPoint.XGUnitNativeBase
Objectlist[12714]: AA 31 00 00  : fDot.IsFlankedByLoc.XGUnitNativeBase
Objectlist[12748]: CC 31 00 00  : fDot.DoesFlankCover.XGUnitNativeBase
Objectlist[12753]: D1 31 00 00  : fDot.DoesFlank.XGUnitNativeBase 

 

 

 

The variables added in XComGame.upk patch 5 were:

 

 

Objectlist[12715]: AB 31 00 00  : ReturnValue.GetBestFlankingDot.XGUnitNativeBase
Objectlist[12716]: AC 31 00 00  : Loc.GetBestFlankingDot.XGUnitNativeBase
Objectlist[12717]: AD 31 00 00  : GetBestFlankingDot.XGUnitNativeBase

Objectlist[14683]: 5B 39 00 00  : kVisibleUnit.InitMoveManeuver.XComAlienPod
Objectlist[14688]: 60 39 00 00  : kUnit.IsPathVisible.XComAlienPod
Objectlist[14689]: 61 39 00 00  : vLoc.IsPathVisible.XComAlienPod
Objectlist[14690]: 62 39 00 00  : nAliens.IsPathVisible.XComAlienPod
Objectlist[14691]: 63 39 00 00  : iAlien.IsPathVisible.XComAlienPod
Objectlist[14692]: 64 39 00 00  : ReturnValue.IsPathVisible.XComAlienPod
Objectlist[14693]: 65 39 00 00  : kVisibleUnit.IsPathVisible.XComAlienPod
Objectlist[14694]: 66 39 00 00  : vHiddenLoc.IsPathVisible.XComAlienPod
Objectlist[14695]: 67 39 00 00  : vPodLoc.IsPathVisible.XComAlienPod
Objectlist[14696]: 68 39 00 00  : IsPathVisible.XComAlienPod

Objectlist[14702]: 6E 39 00 00  : bHasVisibleUnit.WalkPodToLocation.XComAlienPod
Objectlist[14713]: 79 39 00 00  : ReturnValue.WalkPodToLocation.XComAlienPod
Objectlist[14752]: A0 39 00 00  : bForceRecalc.GetPodLocation.XComAlienPod
Objectlist[14958]: 6E 3A 00 00  : bForceRecalc.UpdatePodLocation.XComAlienPod
Objectlist[37902]: 0E 94 00 00  : kUnit.IsEnemyUnitVisibleFromPos.XGPlayer
Objectlist[37903]: 0F 94 00 00  : bVisible.IsEnemyUnitVisibleFromPos.XGPlayer
Objectlist[37910]: 16 94 00 00  : bCheckBothWays.IsEnemyUnitVisibleFromPos.XGPlayer
Objectlist[46870]: 16 B7 00 00  : bCheckBothWays.IsPathVisible.XGUnit 

 

 

 

Four local variables fDot were removed and a new function XGUnitNativeBase.GetBestFlankingDot was added. This new function is a native function with a float return value and a single parameter, Loc. kVisibleUnit was added to XComAlienPod.InitMoveManuever. A new function was added to XComAlienPod named IsPathVisible. Other new variables (and someReturnValues) appear to be addressing whether an alien's path is visible to an enemy.

 

------------------------

 

I've been trying to figure out how to update references in before/after hex and have come to the conclusion that doing global search/replace of (old reference)->(new reference) won't work. Every single integer value from 1 to ~50,000 is a valid reference, which makes the incidence of incorrect replacements too high.

For example, in XComGame.upk:

Objectlist[   44]: 2C 00 00 00  : EPawnType.XGGameData

is a valid reference. However blind searching for 2C 00 00 00 in the list of Long War changes finds the occurrence in:

• 19 19 1B 9C 2C 00 00 00 00 00 00 16 09 00 EB 46 00 00 00 01 EB 46 00 00 0A 00 5D 02 00 00 00 1B 01 11 00 00 00 00 00 00 16

However the actual references in this hex are:

• 19 19 1B 9C 2C 00 00 00 00 00 00 16 09 00 EB 46 00 00 00 01 EB 46 00 00 0A 00 5D 02 00 00 00 1B 01 11 00 00 00 00 00 00 16

In this case this hex actually corresponds to the code: World().m_kFundingCouncil.HasActiveCampaign())

• 9C 2C 00 00 is the namelist index for "World"
• EB 46 00 00 is the objeclist index for the variable m_kFundingCouncil (it is listed twice -- once as a ReturnValue and once as a variable)
• 5D 02 00 00 is the objectlist index for the ReturnValue of HasActiveCampaign
• 01 11 00 00 is the namelist index for "HasActiveCampaign"

The two options I see for enabling more-automated updating for patches are:

1) Basically building a decompiler, duplicating what UE Explorer already does

2) Adding meta-data to assist in identifying references so they can be updated

 

Option 1 seems like far too much work, so I'm going to focus on option #2. :wink:

 

Right now there are two active tools for doing hex replacement, so it would be nice to maintain some compatibility with both of them:

 

1) ToolBoks

 

Bokauk's proposed formatting (copied from earlier in the thread) looks like:

NAME=Disable Friendly Fire (forward-compatible)
AUTHOR=bokauk
DESCRIPTION=New ToolBoks Custom Mod example to demonstrate forward-compatible mods by using variable names.

Compatible with XCOM Enemy Unknown versions:
 - All versions?


LOCATION=XComGame\CookedPCConsole\XComGame.upk>XGUnit.AddTargetsInSight
OFFSET=153
INSTALL_HEX=
96

{ .Length, LocalVariableToken }
36 00

(arrVisibleTargetsToUse) { Variable names inside of parentheses will be converted into hex by ToolBoks }

It uses "<TAG> = " identifiers to set contexts.

 

Advantage is a machine-readable yet very human-readable format.

Disadvantage with this format is the OFFSET=requirement, as upk offsets usually change with a patch.

 

2) XCOMModHelper

This is a newer tool that allows direct hex replacements in the executable as well as upk files. Current formatting (from github) is:

<?xml version="1.0" encoding="utf-8" ?>
<PatcherConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <BackupDirectory>XEW\Backup</BackupDirectory>
  <DecompressedUPKOutputDirectory>XEW\UpkUnpacked</DecompressedUPKOutputDirectory>
  <PatchTargets>
    <PatchTarget>
      <TargetPath>XEW\XComGame\CookedPCConsole\XComGame.upk</TargetPath>
      <IsUPKFile>false</IsUPKFile>
      <Patches>
        <PatchEntry>
          <Description>Unlock Second Wave Options</Description>
          <FindValue>   07 B4 00 81 19 19 2E 98 69 00 00 19 12 20 36 FE FF FF 0A 00 9D F9 FF FF 00 1C DD FB FF FF 16 0A 00 9C F9 FF FF 00 1B A0 3A 00 00 00 00 00 00 16 09 00 F4 68 00 00 00 01 F4 68 00 00 15 00 01 6A 00 00 00 1B 9D 44 00 00 00 00 00 00 38 3D 00 CB 1C 00 00 16 16 06 7F 03</FindValue>
          <ReplaceValue>0B 0B 0B 81 19 19 2E 98 69 00 00 19 12 20 36 FE FF FF 0A 00 9D F9 FF FF 00 1C DD FB FF FF 16 0A 00 9C F9 FF FF 00 1B A0 3A 00 00 00 00 00 00 16 09 00 F4 68 00 00 00 01 F4 68 00 00 15 00 01 6A 00 00 00 1B 9D 44 00 00 00 00 00 00 38 3D 00 CB 1C 00 00 16 16 0B 0B 0B</ReplaceValue>
        </PatchEntry>
      </Patches>
    </PatchTarget>
  </PatchTargets>
</PatcherConfiguration>

This uses a full XML-based specification for contexts.

 

It uses before/after hex strings instead of offset locations, which is more robust for upk patches. However the XML spec is less human-friendly in terms of readability (without an XML viewer, of course ^_^).

 

Currently there is no proposed format for reference specification to allow for replacement.

 

--------------------------------

 

Open to any thoughts on how to specify hex code with reference meta-data clues.

 

[wghost81]

As I wrote in my previous post, I've focused on "meta-data" approach in my attempts of repairing modded functions. I tried to use a list of human-readable variable/function names, but ran into similar problems.

 

Instead of working with entire upk, I try to work with one binary function, extracted from the upk. If it can be repaired, it then can be easily integrated into new upk with any of the existing tool, so, as it seems to me, this approach preserves compatibility.

 

To make things interesting: in EW some object changed their owners. Or owners names. Anyway, name like XGStrategyActor.World is no longer valid in EW, it is XGStrategyActorNativeBase.World instead. So straightforward mapping by object full name will not work when moving from EU to EW.

 

To find and replace object references correctly, we need to know not only the object's name/reference, but also how it is used in the actual code. It may require at least partial code decompilation-compilation, so it won't be easy.

 

Right now it seems that with full list of objects in hands it will be easier to do this manually. :smile:

Edited November 22, 2013 by wghost81

[Bertilsson]

Please correct me if wrong but wouldn't the following approach be feasible?

 

1. Export object code using UE Explorer (results in files like XGPlayer.uc)

2. (New tool) Isolate the functions (results in files like XGPlayer.LoadSquad.txt)

3. (New tool) Export all functions as binary files (results in files like XGPlayer.LoadSquad.bin)

4. Repeat 1-3 for new both old (modded function) and patched upk (results in folder A and B)

5. (New tool) compare all .txt files to isolate functions which are not the identical in OldModdedCode and NewPatchedCode and copy those to new folders (C and D)

6. Manually get rid of files in the new folders which are not relevant to the mod.

 

The result so far should be two folders with the functions that needs to be converted.

 

For each function

7. Put the first 48 bytes in a new file from new patched unmodded bin-file into an new file.

8. Add bytes 49 all the way to 53-token from old modded bin

9. Add all junk after 53 from new new patched unmodded bin

10. Do lookup and replace on everything that looks like references in the object-code files.

 

Sure, there are A LOT of steps involved but it seems fairly doable... to me... The bin files can then be imported manually one by one via UE Explorer or for more brave approach replaced one directly in the upk by searching for the new unmodded bin data and replacing it with new modded bin data...

[wghost81]

Steps 1-6 seem redundant, as modder should know what exactly he modified. Problem, as Amineri explained, is in step 10: not everything that looks like reference is a reference. Furthermore, function header contains some references too and those may change as well.

 

The best approach I can think of right now will be to search for an objects in context rather than just references. For example, Game() function call looks like

1B 4C 0E 00 00 00 00 00 00 16

Object reference for XGStrategyActor.Game is "9D 00 00 00", but since it's a virtual function, it uses namelist index instead, which is "4C 0E 00 00". That's OK, since we can determine function type from tables, but constructions like Game().GetDifficulty() look like

19 1B 4C 0E 00 00 00 00 00 00 16 0A 00 43 41 00 00 00 1B F6 0E 00 00 00 00 00 00 16

This includes two virtual function calls with object reference (which is XGStrategy.GetDifficulty.ReturnValue) in between. So we have to at least take into account object type. But, in this example "00 43 41 00 00" is reference to local variable from function XGStrategyActor.Game, not from the function Game() is called from.

 

Sorry if I can't explain this clearly, my english is not very good. :smile: But the problem is: to determine what to replace we need to analyze the code, hence make a decompiler. And if we try to use complex constructions as meta-data, we'll need a compiler. :smile:

Edited November 22, 2013 by wghost81

[Bertilsson]

The reason for step 1-6 was to get sets of bin and object code files with least amount of fuzz (taking long war as example we are talking about hundreds of changed functions)

 

In object code dumped from UE Explorer you don't need a compiler to identify what is an reference or not if you have access to a look-up function which can provide you with the answer :smile:

 

Break things down into words (but keep words with "." together) and look them up one by one.

 

UE core functions such as Vect should probably be ignored.

 

If going for fully automatic a few extra steps could be added to export new object code files from UE Explorer to see which functions remained identical after the changes were done (if they are not then something went wrong, like failure to update a reference or mistranslated and must be manually checked).

Edited November 22, 2013 by Bertilsson

[wghost81]

Break things down into words (but keep words with "." together) and look them up one by one.

Try to look for Game().GetDifficulty() for example. :smile:

 

If you're looking through UE Explorer decompiled code and trying to find it's byte-code in binary function file, you definitely need a compiler. :smile:

 

OK, some examples. Say, we're looking for arrTypes variable. There are many of those in objectlist table. Well, we know where its came from, so we add a function name to this and happily identify a correct one. Then we try to do search and replace part, but there is no guarantee there won't be some virtual function which just happens to have exactly same namelist index. And if we replace that index in assumption that it's an objectlist index, we'll break a code. But. With some more work we can identify object type: either it's a local variable, function parameter or something and look for it's code in context (like 00 XX XX XX XX for locals). But with Game().GetDifficulty() construction we still miss a return value reference if we just break this into two functions. And if we don't we need to parse this string and compile it, to know what we're looking for.

Edited November 22, 2013 by wghost81

[EliotVU]

I've been called for help, although I'm not a fan of re-doing what's already done. Not sure what exactly is needing my help but nonetheless I noticed and agree that my library lacks documentation. So I have started abit of documentation on its API, you can read about it on github

[Amineri]

I've been called for help, although I'm not a fan of re-doing what's already done. Not sure what exactly is needing my help but nonetheless I noticed and agree that my library lacks documentation. So I have started abit of documentation on its API, you can read about it on github

 

I agree 100% that using the existing UELib.dll would be a lot more productive.

 

Bertilsson and I discussed this early on before I even posted anything, but neither of us knew how to use it. Looked at the documentation and already can see many new possibilities!

 

For me at least tracing through all of this stuff has been at least a educational exercise.

[Amineri]

Regarding the mechanics of actually applying patches :

 

Currently the two methods are:

1) Use before/after hex strings to search & replace

This has the advantage of being more robust to smaller patches that shift file position, as well as built-in error checking (if the before hex isn't found the mod apply tool can throw an error).

 

The big disadvantage is slow speed, having to search through an entire large upk for a large number of S&R operations can be quite slow.

 

2) Use file position and replacement hex

The primary advantage of this method is greatly improved speed.

 

However a patch that adds any content will shift file positions resulting in incorrect hex changes. Also there is little built-in error checking.

 

------------------------

 

The majority of hex changes made are to function code (although I do sometimes change other things like default values, function names, etc).

 

I think I see a 3rd alternative that has closer to the speed of option 2 while retaining the robustness of option 1.

 

By providing a Class/Function context for a hex change, the correct scope of the file hex can be looked up from the object table.

 

For example, including a tag including:

{XGUnit.AddStatModifiers}

defining the function in which the hex to be applied allows the following:

• In patch 5 EU, this string can be used to identify object index 46295 = 0xB74D
• In patch 4 EU, this string would identify object index 46908 = 0xB73C

 

Using this object entry, integers 8 & 9 identify the file size and file position of the function.

• In patch 5 EU, the file size is 0x21D and the file position is 0x913342
• In patch 4 EU, the file size is 0x21D and the file position is 0x9126B9

With this information the scope of a before/after search and replace function would be drastically reduced, resulting in times comparable to using direct file positions, but with the advantages of S&R in terms of robustness.

 

 

-------

 

For more unusual hex changes, some other mechanism would be required.

1. Changing a variable name actually involves changing the string in the namelist
2. Changing a variable type involves changing byte 0 of the variable's object in the objectlist
3. Changing the static array size of a variable involves changing the variable object referenced by the objectlist
4. Default Properties live in a special object. For example the XGUnit Default Properties are inside the object named Default__XGUnit (with 2 underscores).

Changing things related to 3 and 4 would use the same system as for functions.

 

For 3 a tag like :

{XGunit.DrawRangesForMedikit.I}

would uniquely identify the variable object scope file.

 

For 5 a tag like :

{Default__XGUnit}

would uniquely identify the default properties scope in the file.

 

Changing things related to 1 and 2 requires changing the object or namelist entry directly, instead of the file position referenced by the objectlist entry.

[Bertilsson]

 

Break things down into words (but keep words with "." together) and look them up one by one.

Try to look for Game().GetDifficulty() for example. :smile:

 

If you're looking through UE Explorer decompiled code and trying to find it's byte-code in binary function file, you definitely need a compiler. :smile:

 

OK, some examples. Say, we're looking for arrTypes variable. There are many of those in objectlist table. Well, we know where its came from, so we add a function name to this and happily identify a correct one. Then we try to do search and replace part, but there is no guarantee there won't be some virtual function which just happens to have exactly same namelist index. And if we replace that index in assumption that it's an objectlist index, we'll break a code. But. With some more work we can identify object type: either it's a local variable, function parameter or something and look for it's code in context (like 00 XX XX XX XX for locals). But with Game().GetDifficulty() construction we still miss a return value reference if we just break this into two functions. And if we don't we need to parse this string and compile it, to know what we're looking for.

 

Ok, firstly my technical skill in this area is _very_ limited so please try to have a little extra patience if I have misunderstand some of the finer points.

 

But in the case of Game().GetDifficulty()...

 

The reason I wanted to keep "." words together was because I did realize they need special handling compared to references without any "." in the decompiled object code.

 

From my very limited perspective I view Game().GetDifficulty() like this:

Probably19 + ReferenceToGame() + SomeMoreOrLessFixedLengthJunkThatAmineriTendsToEndUpHelpingMeWithAfterIFailAFewTimes + ReferenceToGetDifficulty() + 16

 

It wouldn't be necesarry to imediately try to find the entire hex string... But it would be necessary to figure out how to logically find it and later to replace parts of it (for example memory size should be left alone while return value would have to be updated).

 

As soon as the entire string has been located I would assume a lot of deductions could be made without necessarily building a compiler...

 

And I'm guessing looking for 19orEquivialent + lefthandreference + anything + righthandreference + 16 with the logic to select the possible match with shortest length would go a really long way for figuring out exactly how the entire byte string looks and make it possible to replace any occurance of that string provided that the inner part can be corrected.

 

Then there are of course more complex scenarios... But I'm not looking for a perfect tool, I would be satisfied with a tool that can handle 50% semi-automatically early on and if it could later handle 90% and let me know which other 10% failed I would be perfectly happy as it would have saved 90% of the work.

 

And yes I do realize there is a huge difference in Game().GetDifficulty() and class'Engine'.static.GetEngine().GetContentManager()).GetColorPalette(0)

 

And naturally it would be very interesting to find in which aspects Eliot.UELib.dll can make things possible or simpler.

If for example it could be used to decompile the entire functions into almost atoms and just replace them on high level from one function to the other :smile:

 

-----

 

@Amineri

On the topic of absolute position values (toolboks style) vs search & replace values...

 

If it is possible to identify in which function a given memory position resides... and what the offset for that function is in the old and the new version of the upk... Then would it not also be very simple to calculate the offset between the functions and adjust the absolute position accordingly?

 

In the scenario where references change in a patch it would then be very straight forward to identify the broken code... The search and replace value would pretty much have to be converted to absolute position via the old upk before it could be located in the new upk...

 

And when it comes to XCom EU. The 4 first patches... Were any of them friendly in nature where references did not change or were they all resulting in new references? (I don't know, but am sort of curious if the advantage of search & replace strategy really is relevant for this specific game) :)

Edited November 22, 2013 by Bertilsson