Long War mod discussion

[osito2dancer]

Does "Offline Steam" still work? I always do both phone-home and offline steam to play X-Com, since i dont trust that they just use 2 ips to balance the whole upload infrastructure. Hope it doesnt affect my gameplay, i dont want to update XCOM or buy EW just yet :(

[johnnylump]

I was just able to start up my unpatched LW version (hosts file fix + no auto updates), but it's possible I'm waiting in a queue for the undesired patch, I suppose.

 

Sorry everyone, all I can ask is for you to be patient.

Edited November 12, 2013 by johnnylump

[Amineri]

Sorry I won't be able to help with questions about not updating EU.

 

I went ahead and got EW so I can start rooting around the new code ASAP. I believe it is possible (I did it once) to launch XCOM without even using Steam, by directly launching the XComGame.exe. The major downside I saw is that it then tries to load Saves from a new location in the XCom-Enemy-Unknown/Saves folder instead of from My Games.

 

I launch Victoria 2 (which I bought/installed through Steam) directly all the time, although I haven't done so often with XCOM.

 

In terms of the expansion it looks as though code-wise EU and EW are being kept structurally distinct which means that creating an updated EU-only version of Long War (which should work for both people who do and don't have the expansion) should be relatively straightforward.

[zyllos]

Anything I could possibly help with?

 

I won't touch the base game so I won't be playing until this mod is officially able to be used with EW.

 

What files/code/ect is still needed to be found?

[Amineri]

Anything I could possibly help with?

 

I won't touch the base game so I won't be playing until this mod is officially able to be used with EW.

 

What files/code/ect is still needed to be found?

 

Thanks for the offer!

 

To a certain degree, with respect to EW, we just "don't know what we don't know". The overall structure of the code has remain mostly unchanged (thankfully). However so many new mechanics have been spliced in that we're having to assess what all is new.

 

A fair bit of big stuff and some little stuff too.

 

At this point the biggest amount of work is simply updating for the (minor) Enemy Unknown patch. Basically all of the code-wise changes should be unchanged (code as in what UE Explorer kicks out). However jump offsets and function/variable references have changed so the before/after hex replacements don't work any more.

 

In the Miscellaneous files section of Long War is the 'Long War 2 hex and localization changes' file. It contains basically every hex change made for Long War. Currently it's for the 2.12 beta4, which in terms of hex/localization changes is identical to 2.12 beta5 and 2.12 release.

 

Most (but not all) of the changes refer to the class.function that is being changed. In that function in the XCOM:EU with patch 5 should be very similar code but with function/variable references changed. The challenge is finding the new "before" hex, and then generating new "after" hex with correct references.

 

The file has 3000 lines and is ~780k, so there's a fair number of changes.

 

Any help with this would be much appreciated.

[zyllos]

 

Anything I could possibly help with?

 

I won't touch the base game so I won't be playing until this mod is officially able to be used with EW.

 

What files/code/ect is still needed to be found?

 

Thanks for the offer!

 

To a certain degree, with respect to EW, we just "don't know what we don't know". The overall structure of the code has remain mostly unchanged (thankfully). However so many new mechanics have been spliced in that we're having to assess what all is new.

 

A fair bit of big stuff and some little stuff too.

 

At this point the biggest amount of work is simply updating for the (minor) Enemy Unknown patch. Basically all of the code-wise changes should be unchanged (code as in what UE Explorer kicks out). However jump offsets and function/variable references have changed so the before/after hex replacements don't work any more.

 

In the Miscellaneous files section of Long War is the 'Long War 2 hex and localization changes' file. It contains basically every hex change made for Long War. Currently it's for the 2.12 beta4, which in terms of hex/localization changes is identical to 2.12 beta5 and 2.12 release.

 

Most (but not all) of the changes refer to the class.function that is being changed. In that function in the XCOM:EU with patch 5 should be very similar code but with function/variable references changed. The challenge is finding the new "before" hex, and then generating new "after" hex with correct references.

 

The file has 3000 lines and is ~780k, so there's a fair number of changes.

 

Any help with this would be much appreciated.

 

 

Ok, now I see the "overall" issue, I think.

 

Bear with me, while I might have a degree in Computer Science, typically, I write code then compile instead of decompile (actually, I have never done this before) then modify.

 

But from what I gather, using this UE Explorer, you can decompile the EXE (EDIT: ...I mean the .UPKs, which are apparently like .DLL files utilized by the XCom EXE) file into readable code, modify the function parameters (as you can't physically change the functions themselves) and see how that effects the binary afterwards. Then, you have to match these references so you can get the code to perform "new functionality"?

 

Am I even close?

 

I will take time this weekend and try to understand the hex changes in what they are attempting to accomplish. Hopefully by the end of the weekend, I can actually be of some help. Just like adding a new team member to a development group...basically be useless and will take up a lot of resources/time to get up to speed.

Edited November 14, 2013 by zyllos

[Amineri]

I suppose in a way .upk files are like .dll files in terms of extensibility at runtime. However in a way they are also like java files.

 

.upk I think stands for unreal package. This is compiled hexcode that is designed to be interpreted by the Unreal Engine, which is really a virtual machine optimized for displaying and animating 3D environments (with physics, shooting and all that). .upks aren't really quite as machine independent as something like java because 3D shooters are much more processing intensive, so there is a 'cooking' process which performs some low level optimizations on the upk hex for a particular hardware platform (stuff like little-vs-big endian). So the code gets cooked slightly differently for PC vs XBOX vs PS3, for example.

 

There is also a part of the code that is "native" code, which is an executable designed to run specifically on the hardware/OS of the target platform. The Unreal Engine is designed to be able to make function calls back and forth to/from native/unreal code. Possible callbacks are determined at compile/link time. So the game launches with the executable but the 'thread of execution' can pass into the upk code quite easily.

 

.upks also contain a lot of art assets. Most of the upk data for XCOM is art assets. 3D meshs, textures, animations, and sound data. This is all packed/cooked into upks. Mapfiles are also upks. In Unreal Engine 3 there is also a movie player (plays .bik files) and a flash/actionscript interpreter which allows running embedded actionscript (they are projected onto a 3D surface in-game).

 

So far we've done extremely limited things with regard to the executable (pretty much just stuff like unlocking the developer console, disabling the SHA checks in EU, and force-loading config data from files). Most of the changes we've made have been to the Unreal script, although we've made some progress in modding actionscript as well (allowing for limited UI changes).

 

Access is done via using UE Explorer to decompile the hex code. Fortunately the namelist in Unreal includes a text field, which allows the original variable/function names to be re-constructed on decompile (otherwise it would be near impossible to understand). Using the decompiler we can understand what the unrealscript portion of the game is doing.

 

Unfortunately there is no compiler. Creating such a compiler would pose some tricky issues, in particular having access to the namelist references (which in theory could be read from the upk). Currently my workflow is to:

0) Decompile the upk and find the section I'm interested in

1) Decipher the decompiled code functionality

2) Decide on changes desired

3) Write high level C++ style target code

4) Manually convert each line of target code into hex

5) Replace original hex with new hex

6) Decompile new hex to verify correctness

7) Iterate 4-7 until new code runs as desired

 

It's definitely rather clunky and is slower than just writing code and running the compiler. Basically the (virtual) machine code is created by hand, and there is no IDE which makes debugging more tedious. But it's the best we have so far.

[Amineri]

To give you a better idea of what these changes entail, here's some example hex code.

 

This is from the function XComStrategyGame.upk / XGSoldierUI.UpdateHeader. The modlet is the one that adds the soldier's XP and move stats to the display in the barracks.

 

In Enemy Unknown Patch 4 one change is the following line/hex:

m_kHeader.txtSpeed.StrValue = (m_kHeader.txtSpeed.StrValue $ "+") $ string(aModifiers[3])
0F 35 DD F9 FF FF 6C FA FF FF 00 00 35 47 15 00 00 4E 15 00 00 00 01 01 80 3F 00 00 70 70 35 DD F9 FF FF 6C FA FF FF 00 00 35 47 15 00 00 4E 15 00 00 00 01 01 80 3F 00 00 1F 2B 00 16 38 53 1A 24 03 00 FA 3F 00 00 16 

In Enemy Within (in the XEW folder) the change is instead :

m_kHeader.txtSpeed.StrValue = (m_kHeader.txtSpeed.StrValue $ "+") $ string(aModifiers[3])
0F 35 0C F9 FF FF B9 F9 FF FF 00 00 35 49 1A 00 00 50 1A 00 00 00 01 01 7F 4C 00 00 70 70 35 0C F9 FF FF B9 F9 FF FF 00 00 35 49 1A 00 00 50 1A 00 00 00 01 01 7F 4C 00 00 1F 2B 00 16 38 53 1A 2C 03 00 02 4D 00 00 16 

Hex values such as 0F (for assignment), 35 (for struct construction), 19 (for context construction), and 1B (for virtual function definition) remain unchanged.

 

However all of the 4-byte hex references for things such as aModifiers, m_kHeader, txtSpeed, etc are all different.

[zyllos]

That is the exact explanation I needed.

 

I have never actually just compiled C++ code then looked at the binary to see what the hex values became.

 

Is there a resources of values, as per you explanation of (0F = for assignment, 35 = for struct construction, ...) or is it basically trial and error?

 

Mainly, your step (4) - Manually convert each line of target code into hex, would be my hardest part in an attempt to be useful here.

Edited November 14, 2013 by zyllos

[Amineri]

There is definitely a list of what hex values are used for what : http://wiki.tesnexus.com/index.php/Hex_values_XCOM_Modding

 

Quite a bit of other good stuff that has collected there. Full credit to dubiousintent for maintaining the wiki and chiding us into putting up new info there.

 

Just keep in mind that a particular byte value might be used in different ways depending on its position/context.

 

So 0F might be an assignment operator if starting a new command, but 2C 0F would represent the integer 15, for example. 2C is the hex used to declare integer values (but only using a single following byte).

 

Or 07 0F 00, which would represent a conditional jump to offset 0x000F.

 

Fortunately it's not all that bad, as UE Explorer's token view breaks down the hex-to-code on a line-to-line correspondence.