R&D - New Modding Tools

[Drakous79]

Amineri it looks great :)

[dubiousintent]

Go ahead and feel giddy for a bit. You've earned it! This is going to be a great asset.

 

-Dubious-

[wghost81]

Wow. You are one step from writing an actual compiler. Should really think about it. :wink:

[Amineri]

I'm actually fairly close to having a full decompiler (including calculation of memory sizes), but going the other direction and compiling unrealscript is going to be a bit more work -- and I have thought about it :).

 

The plan for the initial beta release is to have the tool that can do the following :

• Basic new/open/save/save as functionality for modfile documents
• Text editor with correct highlighting of unreal bytecode to allow visual confirmation that things are working as intended
• Ability to update (via menu command) all references in a document from the source upk (specified by filename and GUID) to a user-designated target upk

And that's basically it.

 

Things that I definitely want to add soon that I think would be really helpful:

• Convert hex references to/from named references (e.g. ||ReturnValue@CollectArtifactsFromDeadAliens@XGSummaryUI||)
• Ability to apply/revert hex to a upk to facilite testing
  • Would include dashboard indicator to inform user whether hex is applied or not
• Ability to export to common mod installer formats (ToolBoks & XComModHelper)
• Display of memory sizes for unreal bytecode
• Jump Offset (both absolute and relative) recalculation tool
• XML-based project file support to contain/manage multiple modfiles

Eventually perhaps a small assistant-type tool to help in the construction of unreal byte code might make it in as well.

 

However, fairly soon I foresee switching over from developing this tool to doing more actual modding :)

 

---------------

 

One final thing I wanted to mention -- this tool actually has nothing at all (as far as I'm aware) that is specific to XCOM. The upk parsing / highlighting should be just as effective for modding any other Unreal Engine 3 game. Of course I haven't testing this :D

[troch22]

Very impressive. .

[Amineri]

So the best I've been able to do in terms of "version agnostic" is conversion of a modfile to something like so :

 

 

 

MODFILEVERSION=3
UPKFILE=XComGame.upk 
GUID= 01 E9 EB 29 23 F4 DB 4F A8 2B 8E 46 A7 25 E5 D6 // EU patch 4
FUNCTION=ApplyActionCost@XGAbilityTree
// Increase max pod size Mod
// Author: Amineri 
[BEFORE_HEX]
[code]
// PlayerIndex = 0;   PlayerIndex@InitAlienLoadout@XGBattleDesc
0F 00 {|PlayerIndex@InitAlienLoadoutInfos@XGBattleDesc|} 25 
[/CODE]
[/BEFORE_HEX]

// line with parse error -- incorrect trailing 01 value
[AFTER_HEX]
[code]
// PlayerIndex = 1;   PlayerIndex@InitAlienLoadout@XGBattleDesc
0F 00 10 A0 00 00 26 01
[/CODE]
[/AFTER_HEX]


[BEFORE_HEX]
[code] // parsable unrealhex -- corresponds to full lines of code -- allows operand decoding
//iCost = XComGameReplicationInfo(class'Engine'.static.GetCurrentWorldInfo().GRI).m_kGameCore.GetAmmoCost(kAbility.m_kWeapon.GameplayType(), kAbility.GetType(), kAbility.m_kUnit.GetPlayer().HasFoundryHistory(10), kAbility.m_kUnit.GetCharacter().m_kChar, kAbility.m_bReactionFire); (259 file, 379 virtual bytes -- 120 extra)
0F 00 {|iCost@ApplyActionCost@XGAbilityTree|} 19 19 2E {|XComGameReplicationInfo|} 19 12 20 {|Core:Engine@Engine|} 0A 00 {|Core:ReturnValue@GetCurrentWorldInfo@Engine@Engine|} 00 1C {|Core:GetCurrentWorldInfo@Engine@Engine|} 16 09 00 {|Core:GRI@WorldInfo@Engine|} 00 01 {|Core:GRI@WorldInfo@Engine|} 09 00 {|m_kGameCore@XComGameReplicationInfo|} 00 01 {|m_kGameCore@XComGameReplicationInfo|} 13 01 {|ReturnValue@GetAmmoCost@XGTacticalGameCoreNativeBase|} 00 1B <|GetAmmoCost|> 00 00 00 00 38 3A 19 19 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 09 00 {|m_kWeapon@XGAbility_Targeted|} 00 01 {|m_kWeapon@XGAbility_Targeted|} 0A 00 {|ReturnValue@GameplayType@XGItem|} 00 1B <|GameplayType|> 00 00 00 00 16 19 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 0A 00 {|ReturnValue@GetType@XGAbility|} 00 1B <|GetType|> 00 00 00 00 16 19 19 19 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 09 00 {|m_kUnit@XGAbility|} 00 01 {|m_kUnit@XGAbility|} 0A 00 {|ReturnValue@GetPlayer@XGUnit|} 00 1B <|GetPlayer|> 00 00 00 00 16 0C 00 {|ReturnValue@HasFoundryHistory@XGPlayer|} 00 1B <|HasFoundryHistory|> 00 00 00 00 24 0A 16 19 19 19 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 09 00 {|m_kUnit@XGAbility|} 00 01 {|m_kUnit@XGAbility|} 0A 00 {|ReturnValue@GetCharacter@XGUnit|} 00 1B <|GetCharacter|> 00 00 00 00 16 09 00 {|m_kChar@XGCharacter|} 00 01 {|m_kChar@XGCharacter|} 19 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 0A 00 {|m_bReactionFire@XGAbility|} 00 2D 01 {|m_bReactionFire@XGAbility|} 16 
[/CODE]
[/BEFORE_HEX]

[AFTER_HEX]
[code]
// iCost = kAbility.GraduatedOdds(0, kAbility, kAbility.m_kUnit.GetPlayer().HasFoundryHistory(10)); (92 file, 128 virtual bytes -- 36 extra)
0F 00 {|iCost@ApplyActionCost@XGAbilityTree|} 19 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 61 00 {|iFactor@GraduatedOdds@XGAbility_Targeted|} 00 1B <|GraduatedOdds|> 00 00 00 00 2C 00 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 19 19 19 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 09 00 {|m_kUnit@XGAbility|} 00 01 {|m_kUnit@XGAbility|} 0A 00 {|ReturnValue@GetPlayer@XGUnit|} 00 1B <|GetPlayer|> 00 00 00 00 16 0C 00 {|ReturnValue@HasFoundryHistory@XGPlayer|} 00 1B <|HasFoundryHistory|> 00 00 00 00 24 0A 16 16 

// null-ops
0B 0B 0B 0B 0B 0B 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 00 {|iCost@ApplyActionCost@XGAbilityTree|} 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 
[/CODE]
[/AFTER_HEX] 

 

 

 

The above file had every reference value detected automatically.

 

Interestingly, the example mod that I chose did not port over to Enemy Within.

 

In EU, foundry upgrades were handled by passing the foundry project list to the tactical game and using it directly, so there is a function HasFoundryProject. This is used above (and in EU) to determine the amount of ammo used. In EW foundry (and other) upgrades are handled by granting a perk to the affected unit. For example, ePerk_Foundry_AmmoConservation = 131. In EW the HasFoundryProject() function has been removed.

 

EU line:

iCost = XComGameReplicationInfo(class'Engine'.static.GetCurrentWorldInfo().GRI).m_kGameCore.GetAmmoCost(kAbility.m_kWeapon.GameplayType(), kAbility.GetType(), kAbility.m_kUnit.GetPlayer().HasFoundryHistory(10), kAbility.m_kUnit.GetCharacter().m_kChar, kAbility.m_bReactionFire);

EW line:

iCost = XComGameReplicationInfo(class'Engine'.static.GetCurrentWorldInfo().GRI).m_kGameCore.GetAmmoCost(kAbility.m_kWeapon.GameplayType(), kAbility.GetType(), kAbility.m_kUnit.GetCharacter().m_kChar.aUpgrades[113] > 0, kAbility.m_kUnit.GetCharacter().m_kChar, kAbility.m_bReactionFire); 

If you look carefully you can see that one parameter changed from kAbility.m_kUnit.GetPlayer().HasFoundryHistory(10) to kAbility.m_kUnit.GetCharacter().m_kChar.aUpgrades[113] > 0.

 

This means that the code itself as a part of the mod was changed and cannot be 100% automatically updated. My current code can identify which references cannot be updated. For the above function this is "HasFoundryHistory" and "ReturnValue@HasFoundryHistory". I'll need to do some experimentation to determine the exact utility of partial updating (e.g. updating those references that can be updated).

 

For the above code when I transformed the names back into EU patch 5 references the hex block was findable. However it (clearly) didn't work for EW release.

 

[dubiousintent]

Actually, I view this as fortuitous news. The earlier such things are discovered, the less frantic alterations are needed later on after everyone has been lulled into complacency that "all is known". See: it's already proving it's usefulness. :laugh:

 

-Dubious-

[Amineri]

Here is the best workflow I've been able to come up with to handle cases where not all references (function or variable) are updated. In the previous example the HasFoundryUpgrade function was removed in the Enemy Within expansion (and instead is now handled via perks directly assigned to units).

 

In this example I convert the mod file shown from EU patch 4 to EW release.

 

Step 1) Convert all reference values to names, using the XComGame_EUpatch4.upk as reference. This results as shown two posts above.

This results in a modfile that is "version agnostic" -- it has no reference values and so is not intrinsically associated with a particular game version. Of course the underlying code structure may have changed.

 

Step 2) Convert all possible names into values for the target upk, in this case using XComGame_EWrelease.upk. This converts all of the names into the values used for that game version. HasFoundryUpgrade() (and it's associated ReturnValue) are not converted since they don't exist in EW.

 

This result in the file:

 

 

MODFILEVERSION=3
UPKFILE=XComGame.upk 
GUID= B1 1A D8 E4 48 29 FC 43 8E C0 7A B0 A3 3E 34 9F // EW release
FUNCTION=ApplyActionCost@XGAbilityTree
// Increase max pod size Mod
// Author: Amineri 
[BEFORE_HEX]
[CODE]
// PlayerIndex = 0;   PlayerIndex@InitAlienLoadoutInfos@XGBattleDesc
0F 00 FF B2 00 00 25 
[/CODE]
[/BEFORE_HEX]

// line with parse error -- incorrect trailing 01 value
[AFTER_HEX]
[CODE]
// PlayerIndex = 1;   PlayerIndex@InitAlienLoadoutInfos@XGBattleDesc
0F 00 FF B2 00 00 2C 01 
[/CODE]
[/AFTER_HEX]


[BEFORE_HEX]
[CODE] // parsable unrealhex -- corresponds to full lines of code -- allows operand decoding
//iCost = XComGameReplicationInfo(class'Engine'.static.GetCurrentWorldInfo().GRI).m_kGameCore.GetAmmoCost(kAbility.m_kWeapon.GameplayType(), kAbility.GetType(), kAbility.m_kUnit.GetPlayer().HasFoundryHistory(10), kAbility.m_kUnit.GetCharacter().m_kChar, kAbility.m_bReactionFire); (259 file, 379 virtual bytes -- 120 extra)
0F 00 69 8C 00 00 19 19 2E 2D 32 00 00 19 12 20 36 FE FF FF 0A 00 9E F9 FF FF 00 1C DE FB FF FF 16 09 00 5C F9 FF FF 00 01 5C F9 FF FF 09 00 1F 32 00 00 00 01 1F 32 00 00 13 01 C4 10 00 00 00 1B 26 37 00 00 00 00 00 00 38 3A 19 19 00 6A 8C 00 00 09 00 AE D2 00 00 00 01 AE D2 00 00 0A 00 CC B9 00 00 00 1B 8B 36 00 00 00 00 00 00 16 19 00 6A 8C 00 00 0A 00 61 89 00 00 00 1B 95 3B 00 00 00 00 00 00 16 19 19 19 00 6A 8C 00 00 09 00 27 89 00 00 00 01 27 89 00 00 0A 00 1F C6 00 00 00 1B 68 3A 00 00 00 00 00 00 16 0C 00 {|ReturnValue@HasFoundryHistory@XGPlayer|} 00 1B <|HasFoundryHistory|> 00 00 00 00 24 0A 16 19 19 19 00 6A 8C 00 00 09 00 27 89 00 00 00 01 27 89 00 00 0A 00 B3 C7 00 00 00 1B 8F 37 00 00 00 00 00 00 16 09 00 E8 B5 00 00 00 01 E8 B5 00 00 19 00 6A 8C 00 00 0A 00 13 89 00 00 00 2D 01 13 89 00 00 16 
[/CODE]
[/BEFORE_HEX]

[AFTER_HEX]
[CODE]
// iCost = kAbility.GraduatedOdds(0, kAbility, kAbility.m_kUnit.GetPlayer().HasFoundryHistory(10)); (92 file, 128 virtual bytes -- 36 extra)
0F 00 69 8C 00 00 19 00 6A 8C 00 00 61 00 03 8A 00 00 00 1B 65 3C 00 00 00 00 00 00 2C 00 00 6A 8C 00 00 19 19 19 00 6A 8C 00 00 09 00 27 89 00 00 00 01 27 89 00 00 0A 00 1F C6 00 00 00 1B 68 3A 00 00 00 00 00 00 16 0C 00 {|ReturnValue@HasFoundryHistory@XGPlayer|} 00 1B <|HasFoundryHistory|> 00 00 00 00 24 0A 16 16 

// null-ops
0B 0B 0B 0B 0B 0B 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 
[/CODE]
[/AFTER_HEX] 

 

 

 

At this point an actual person has to step in to adjust the code.

 

My nexts steps would be:

 

Step 3) Retrieve the new byte code for the new line(s) from UE Explorer. This would replace the previous [bEFORE] block.

 

Step 4) Update the kAbility.m_kUnit.GetPlayer().HasFoundryHistory(10) section of my new target hex with the hex construction for retrieving the perk from the unit. The other sections of my new hex would remain unchanged.

 

This is rather akin to the merge process that has to be undertaken when merging two changed files in a repository. Here the two "coders" are the modder (who made a branch off of EU patch 4, e.g. Long War 2.12) and Firaxis (who made a separate branch off of EU patch 4, e.g. Enemy Within release).

 

What the tool will do is handle updating the underlying reference changes as best possible from one game version to another, leaving only the higher-level conflict resolution to the modder(s).

 

 

-----------------------

 

As a side note :

 

Being able to retrieve the names of every object is going to be really helpful when debugging manually created bytecode.

 

For example, in the line :

// iCost = kAbility.GraduatedOdds(0, kAbility, kAbility.m_kUnit.GetPlayer().HasFoundryHistory(10)); (92 file, 128 virtual bytes -- 36 extra)
0F 00 {|iCost@ApplyActionCost@XGAbilityTree|} 19 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 61 00 {|iFactor@GraduatedOdds@XGAbility_Targeted|} 00 1B <|GraduatedOdds|> 00 00 00 00 2C 00 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 19 19 19 00 {|kAbility@ApplyActionCost@XGAbilityTree|} 09 00 {|m_kUnit@XGAbility|} 00 01 {|m_kUnit@XGAbility|} 0A 00 {|ReturnValue@GetPlayer@XGUnit|} 00 1B <|GetPlayer|> 00 00 00 00 16 0C 00 {|ReturnValue@HasFoundryHistory@XGPlayer|} 00 1B <|HasFoundryHistory|> 00 00 00 00 24 0A 16 16 

The third named reference is showing as {|iFactor@GraduatedOdds@XGAbility_Targeted|} when it should be {|ReturnValue@GraduatedOdds@XGAbility_Targeted|}

 

This reference is something that doesn't show in decompiled code : kAbility.GraduatedOdds(...) doesn't explicitly show the return value of the function (it's basically bookkeeping that the unrealscript compiler handles), but it something that has be to taken into account by a modder working with the bytecode.

 

The above bug is likely the source of some of the occasionaly glitchiness associated with the ammo mod.

 

Eventually I'd like to have easier access to each reference's name (e.g. right-click or a mouse-hover or something) to help the user debug the hex. Going along with that would be a search function to find references based on name.

[Amineri]

I think we're close to having a usable and releasable beta version. With my internal beta I was able to update a line change for the ammo mod (the one that changes the GetAmmoCost call from the native function to a reworked upk function) :

 

 

 

MODFILEVERSION=3
UPKFILE=XComGame.upk 
GUID= B1 1A D8 E4 48 29 FC 43 8E C0 7A B0 A3 3E 34 9F // EW release
FUNCTION=ApplyActionCost@XGAbilityTree
// Increase max pod size Mod
// Author: Amineri 
[BEFORE_HEX]
[CODE] // parsable unrealhex -- corresponds to full lines of code -- allows operand decoding
//iCost = XComGameReplicationInfo(class'Engine'.static.GetCurrentWorldInfo().GRI).m_kGameCore.GetAmmoCost(kAbility.m_kWeapon.GameplayType(), kAbility.GetType(), kAbility.m_kUnit.GetCharacter().m_kChar.aUpgrades[113] > 0, kAbility.m_kUnit.GetCharacter().m_kChar, kAbility.m_bReactionFire)
0F 00 69 8C 00 00 19 19 2E 2D 32 00 00 19 12 20 36 FE FF FF 0A 00 9E F9 FF FF 00 1C DE FB FF FF 16 09 00 5C F9 FF FF 00 01 5C F9 FF FF 09 00 1F 32 00 00 00 01 1F 32 00 00 29 01 C4 10 00 00 00 1B 26 37 00 00 00 00 00 00 38 3A 19 19 00 6A 8C 00 00 09 00 AE D2 00 00 00 01 AE D2 00 00 0A 00 CC B9 00 00 00 1B 8B 36 00 00 00 00 00 00 16 19 00 6A 8C 00 00 0A 00 61 89 00 00 00 1B 95 3B 00 00 00 00 00 00 16 97 1A 2C 71 35 54 0F 00 00 58 0F 00 00 00 00 19 19 19 00 6A 8C 00 00 09 00 27 89 00 00 00 01 27 89 00 00 0A 00 B3 C7 00 00 00 1B 8F 37 00 00 00 00 00 00 16 09 00 E8 B5 00 00 00 01 E8 B5 00 00 25 16 19 19 19 00 6A 8C 00 00 09 00 27 89 00 00 00 01 27 89 00 00 0A 00 B3 C7 00 00 00 1B 8F 37 00 00 00 00 00 00 16 09 00 E8 B5 00 00 00 01 E8 B5 00 00 19 00 6A 8C 00 00 0A 00 13 89 00 00 00 2D 01 13 89 00 00 16 
[/CODE]
[/BEFORE_HEX]

[AFTER_HEX]
[CODE]
// iCost = kAbility.GraduatedOdds(0, kAbility, kAbility.m_kUnit.GetCharacter().m_kChar.aUpgrades[113] > 0); 
0F 00 69 8C 00 00 19 00 6A 8C 00 00 61 00 03 8A 00 00 00 1B 65 3C 00 00 00 00 00 00 2C 00 00 6A 8C 00 00 97 1A 2C 71 35 54 0F 00 00 58 0F 00 00 00 00 19 19 19 00 6A 8C 00 00 09 00 27 89 00 00 00 01 27 89 00 00 0A 00 B3 C7 00 00 00 1B 8F 37 00 00 00 00 00 00 16 09 00 E8 B5 00 00 00 01 E8 B5 00 00 25 16 16 

// null-ops
0B 0B 0B 0B 0B 0B 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 00 69 8C 00 00 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 
[/CODE]
[/AFTER_HEX] 

 

 

 

This took me about 8 minutes to update. And that includes have to work around some ugly UPKmodder issues like : (a) can't save the file yet (b) can't apply/revert changes to a upk © can't convert individual reference values to name within the editor, or view reference names for values in the editor.

 

With those improvement the time would be cut further.

[eld10]

This is really impressive work, you guys!