New tool available that recalulates the jump offsets automatically

[Bertilsson]

If you enjoy manually searching for and recalculating memory offsets when you have made changes to a function then you can stop reading now.

 

For anyone else this tool should probably be very interesting:

http://hem.bredband.net/bertrich/XCOM/JumpAssistant.htm

-----------------------------------------------------------------

Edit: The tool has become obsolete and has been replaced with a new offset repair tool which is much better in every way, skip directly to page 3 for discussion/information about the new tool.

-----------------------------------------------------------------

Instruction

1. Get UE Explorer 1.2.3.0 beta (older versions are not compatible)
2. Copy original hex code and View Tokens data from UE Explorer
3. Modify the .upk file using HxD or any other hex editor
4. Copy modified hex code and View Tokens data from UE Explorer
5. Press Generate New Hex code button to receive a new hex code with recalculated jump offsets for header/53 and any 06, 07, 0A, 2F and 58 tokens "not directly affected by your mod". *See Limitations
6. Insert new hex code in the .upk file using HxD or any other hex editor

* Limitations and how it works

• The tool is basically a data miner that looks at the world as a bunch of view token lines with (MemoryLocation/...) [DataStrings] and some modedHexCode with jump tokens in need of recalculation. The old hexcode is pretty much ignored for anything else than storage size verification.
  • If it discovers that there is ModifiedHexCode pointing to a valid Old.MemoryLocation it then tries to find a Modded.Memory location by comparing [DataStrings]. (This is where the 0A 00 00 bug is born, as it looks like a jump token to a valid memory offset 000).
  • For the few situations where multiple possible matches were found it tries to figure out which one is correct by looking at known move distances for other more succesfull matches.
  • If it fails to find 1:1 pairs you will be informed about it.

The tool is intended to save you the trouble of recalculating the jump offsets for things you didn't change.

 

The tool is not intended to fix the things you changed.

 

Example: If there was a jump token reference to (MemoryLocation/...) [DataString] and you changed the [DataString] then tool will not find it.

 

Known bugs

• Incorrect identification of Jump Tokens 0A 00 00: "(0) Token 0A 00 00 at hex storage offset 0 + 133: Target Memory 0x0000 has not changed."
  • Just ignore it for now, as it is extremely unlikely to have any effect on your mod.
• Buggy user interface

 

Missing features not yet implemeted

• Support for header size other than 48 bytes.
• Undo buttons for each individual change added by the tool
• More intuitive and less buggy way to autoexpand smaller modified hex code to original storage size
  • Currently you can do it by inserting modded hex code of smaller size and pressing generate button before step 4 in above instruction.

Any feedback or suggestions not related to the bugs in the user interface is welcome :smile:

Edited October 3, 2013 by Bertilsson

[Bertilsson]

Example feedback from the tool when i tested making a small change inside a for loop in the loadsquad function:

(0) Token 0A 00 00 at hex storage offset 0 + 594: Target Memory 0x0000 has not changed.
(0) Token 0A 00 00 at hex storage offset 0 + 617: Target Memory 0x0000 has not changed.
(0) Token 0A 00 00 at hex storage offset 0 + 997: Target Memory 0x0000 has not changed.
(0) Token 06 85 00 at hex storage offset 0 + 194: Target Memory 0x0085 has not changed.
(0) Token 07 E4 00 at hex storage offset 0 + 134: Target Memory 0x00E4 has not changed.
(0) Token 06 3A 01 at hex storage offset 0 + 1021: Target Memory 0x013A has not changed.
(-8) Token 07 4E 02 at hex storage offset 0 + 279: Target Memory 0x024E has been changed to 0246.
(-8) Token 06 75 03 at hex storage offset 0 + 432: Target Memory 0x0375 has been changed to 036D.
(-8) Token 07 75 03 at hex storage offset 0 + 554: Target Memory 0x0375 has been changed to 036D.
(-8) Token 07 C4 04 at hex storage offset 0 + 831: Target Memory 0x04C4 has been changed to 04BC.
(-8) Token 07 75 05 at hex storage offset 0 + 845: Target Memory 0x0575 has been changed to 056D.
(-8) Token 06 BD 05 at hex storage offset 0 + 842: Target Memory 0x05BD has been changed to 05B5.
(-8) Token 07 CB 05 at hex storage offset 0 + 251: Target Memory 0x05CB has been changed to 05C3.
(-4) Token HD CE 05 at hex storage offset 0 + 40: Target Memory 0x05CE has been changed to 05CA.

After inserting the new code via HxD into the XComGame.upk-file the game ran the new mod without any problem after a total of exactly zero manual offset calculations :smile:

Edited July 27, 2013 by Bertilsson

[dubiousintent]

YES! Very helpful. I'll be adding this to the Tools article (and others as I think of them) immediately.

 

-Dubious-

[Amineri]

I'm still a bit curious as to how the tool decides which is the correct location to jump to.

 

For example, suppose I have the following info from UE Explorer :

(02D/021) [07 02 01 19 00 DD 42 00 00 0A 00 33 24 00 00 00 2D 01 33 24 00 00]
	JIN(34/22) -> C(31/19) -> LV(9/5) -> BV(10/6) -> IV(9/5)
	if(kCountry.m_bSecretPact)

(04F/037) [07 02 01 19 00 DD 42 00 00 0A 00 76 24 00 00 00 1B 24 11 00 00 00 00 00 00 16]
	JIN(34/26) -> C(31/23) -> LV(9/5) -> VF(10/10) -> EFP(1/1)
	if(kCountry.HasSatelliteCoverage())

(071/051) [19 1B E1 00 00 00 00 00 00 00 16 17 00 00 00 00 00 00 1B C7 00 00 00 00 00 00 00 38 3D 19 00 DD 42 00 00 0A 00 5E 24 00 00 00 1B 58 0F 00 00 00 00 00 00 16 00 AB 44 00 00 16]
	C(74/58) -> VF(10/10) -> EFP(1/1) -> VF(52/40) -> CITB(33/24) -> C(31/23) -> LV(9/5) -> VF(10/10) -> EFP(1/1) -> LV(9/5) -> EFP(1/1)
	AI().AddToAssets(byte(kCountry.GetID()), arrAssets)

(0BB/08B) [07 02 01 81 2D 01 2D 2E 00 00 16]
	JIN(15/11) -> NF(12/8) -> BV(10/6) -> IV(9/5) -> EFP(1/1)
	if(!m_bCodeCracked)

(0CA/096) [14 2D 01 2D 2E 00 00 27]
	LB(12/8) -> BV(10/6) -> IV(9/5) -> T(1/1)
	m_bCodeCracked = true

(0D6/09E) [1B 34 14 00 00 00 00 00 00 16]
	VF(10/10) -> EFP(1/1)
	IsCodeActive()

(0E0/0A8) [19 1B 12 22 00 00 00 00 00 00 16 0C 00 00 00 00 00 00 1B 2B 2B 00 00 00 00 00 00 24 04 16]
	C(34/30) -> VF(10/10) -> EFP(1/1) -> VF(12/12) -> BC(2/2) -> EFP(1/1)
	PRES().UIObjectiveDisplay(4)

This has the structure:

if(A)

if(B)

X

if(D)

Y

Z

W

What I want it to look like is :

if(A)
{
   if(B)
   {
      X
      if(D)
      {
         Y
         Z
         W
      }
   }
}

However, the structure could easily be very different depending on how the jumps are set up. For example, just changing the jump offsets it could be:

if(A)
{
   if(B)
   {
      X
   }
}
if(D)
{
   Y
}
Z
W

I'm curious how the tool would tell the difference. Or am I missing the point and that the tool is primarily for changing the size of code inside existing conditionals, assuming that the conditionals themselves don't change?

 

Sorry for my denseness on this point ... ~_~

[Amineri]

Oh, I forgot to include the "before" code. In this case it looks completely different -- the functionality was drastically altered :

 

 

(017/00F) [0F 00 AE 2E 00 00 25]
	L(11/7) -> LV(9/5) -> IZ(1/1)
	iMission = 0

(022/016) [07 3E 01 96 00 AE 2E 00 00 36 19 1B 69 0E 00 00 00 00 00 00 16 09 00 94 2F 00 00 00 01 94 2F 00 00 16]
	JIN(46/34) -> NF(43/31) -> LV(9/5) -> DAL(32/24) -> C(31/23) -> VF(10/10) -> EFP(1/1) -> IV(9/5) -> EFP(1/1)
	if(iMission < GEOSCAPE().m_arrMissions.Length)

(050/038) [07 30 01 9A 19 10 00 AE 2E 00 00 19 1B 69 0E 00 00 00 00 00 00 16 09 00 94 2F 00 00 00 01 94 2F 00 00 09 00 35 37 00 00 00 01 35 37 00 00 2C 06 16]
	JIN(69/49) -> NF(66/46) -> C(62/42) -> DAE(41/29) -> LV(9/5) -> C(31/23) -> VF(10/10) -> EFP(1/1) -> IV(9/5) -> IV(9/5) -> ICB(2/2) -> EFP(1/1)
	if(GEOSCAPE().m_arrMissions[iMission].m_iMissionType == 6)

(095/069) [0F 19 10 00 AE 2E 00 00 19 1B 69 0E 00 00 00 00 00 00 16 09 00 94 2F 00 00 00 01 94 2F 00 00 09 00 30 37 00 00 00 01 30 37 00 00 25]
	L(64/44) -> C(62/42) -> DAE(41/29) -> LV(9/5) -> C(31/23) -> VF(10/10) -> EFP(1/1) -> IV(9/5) -> IV(9/5) -> IZ(1/1)
	GEOSCAPE().m_arrMissions[iMission].m_iDetectedBy = 0

(0D5/095) [19 10 00 AE 2E 00 00 19 1B 69 0E 00 00 00 00 00 00 16 09 00 94 2F 00 00 00 01 94 2F 00 00 26 00 00 00 00 00 00 1B C6 23 00 00 00 00 00 00 1C A3 FD FF FF 20 DB 38 00 00 4A 4A 4A 4A 4A 4A 4A 16 24 13 16]
	C(91/67) -> DAE(41/29) -> LV(9/5) -> C(31/23) -> VF(10/10) -> EFP(1/1) -> IV(9/5) -> VF(38/30) -> FF(26/18) -> OC(9/5) -> NP(1/1) -> NP(1/1) -> NP(1/1) -> NP(1/1) -> NP(1/1) -> NP(1/1) -> NP(1/1) -> EFP(1/1) -> BC(2/2) -> EFP(1/1)
	GEOSCAPE().m_arrMissions[iMission].SetEntity(Spawn(class'XGMissionEntity'), 19)

(130/0D8) [A5 00 AE 2E 00 00 16]
	NF(11/7) -> LV(9/5) -> EFP(1/1)
	++ iMission

(13B/0DF) [06 22 00]
	J(3/3)
	goto J0x22

 

 

[Bertilsson]

I have updated first post to make it a bit more clear what is done.

 

But from a quick look I would expect the tool to resolve all of the changes as long as you did NOT alter the existing jump tokens so that they are no longer pointing to a valid memory position in Old Tokens and the target[dataString] itself was not modified, but merly moved around.

 

Think of it more like this:

Code section A

Code section B

Code section C

Code Section D

Code Section E

 

In your modded version you have removed the code in section B in order to add stuff in section D.

 

Your new code will be:

Code section A

Code section C

Code Section D with modifications

Code Section E

 

The tool will find new locations of any old reference to and from any part of section A, C and E.

It will also find the unmodified parts (on a per view token line [stringValue] basis) in section D, but it will not fix the modified parts inside D and it will also fail to discover any references to them.

 

To find a better answer to the question you could actually make a simple experiment:

1. Enter original code as both old and modified code in the tool and press generate button.

2. The tool will then tell you which jump tokens it found and that nothing was changed with any of them (or you have found a bug).

3. Store the output "nothing was changed yada yada lines" somewhere like notepad.

4. Insert modified code into the tool and press generate button.

5. Compare the output lines from step 3 and 4.

Edited July 27, 2013 by Bertilsson

[Bertilsson]

YES! Very helpful. I'll be adding this to the Tools article (and others as I think of them) immediately.

 

-Dubious-

Thank you!

[Bertilsson]

I'm still a bit curious as to how the tool decides which is the correct location to jump to.

 

 

The really short alternative answer is:

It dumbs down all view token sections to combinations of Location & Data which is truely unique.

As long as Data only occurs once in old tokens and modified tokens then it can always be found by simply searching for it without even looking at location.

 

The original code included a lot of complicated neighbour comparisons to resolve multiple matching target locations in modified view tokens.

 

In the latest code I threw all that out and replaced it with a simple check if any of the multiple possible targets would fit if an already known move distance was added to the locations. In the very special situation where that logic does not work it simply informs the user that it failed to determine which of location X, Y and Z that was correct and ask the user to sort it out on his/her own.

Edited July 27, 2013 by Bertilsson

[Bertilsson]

I just discovered something strange...

 

I just tested to remove all multiplayer code from the LoadSquad-function in XGPlayer.

This resulted in 336 memorybytes shorter code and A LOT of 0B tokens before the end of script token 53.

 

However it also resulted in UE Explorer not showing any token 53 at the end of my new view tokens.

 

I have now added a warning instead of crash when token 53 is missing in the modified view objects.

However the really strange part is that when I manually pretended that the last available view token was 53-token, the game did NOT crash, instead it happily accepted my change to the function.

 

So right now I'm thinking that this only a cosmetical bug that UE E. somehow shows token 53 as 0B when preceded with a bunch of 0B tokens or possibly the result of some weird mistake I did when removing code, which in that case is even more strange that it did not crash the game.

 

if anyone wants to study the phenomenon it happened when I replaced the entire LoadSquad function (offset 8571137) with this:

 

 

C6 94 00 00 50 55 00 00 00 00 00 00 A0 94 00 00 00 00 00 00 00 00 00 00 B1 94 00 00 00 00 00 00 97 03 00 00 8D 7B 00 00 3E 05 00 00 D6 03 00 00 0F 00 A9 94 00 00 36 00 B1 94 00 00 0F 01 E3 93 00 00 1C 3D FC FF FF 20 21 AD 00 00 4A 4A 4A 4A 4A 4A 01 C5 FE FF FF 16 0F 19 01 E3 93 00 00 09 00 30 AC 00 00 00 01 30 AC 00 00 17 54 01 CC 93 00 00 2C 13 16 0F 10 25 01 CC 93 00 00 26 0F 00 A8 94 00 00 26 07 E4 00 82 96 00 A8 94 00 00 2C 13 16 18 16 00 96 00 A8 94 00 00 36 00 B0 94 00 00 16 16 0F 10 00 A8 94 00 00 01 CC 93 00 00 10 00 A8 94 00 00 00 B0 94 00 00 A5 00 A8 94 00 00 16 06 85 00 0F 01 CB 93 00 00 00 AF 94 00 00 0F 01 CA 93 00 00 00 AE 94 00 00 55 00 AD 94 00 00 0C 00 10 25 00 AD 94 00 00 16 40 00 AD 94 00 00 25 26 16 0F 00 A8 94 00 00 25 07 73 04 96 00 A8 94 00 00 00 A9 94 00 00 16 0F 00 A7 94 00 00 10 25 00 AD 94 00 00 0F 00 AB 94 00 00 1C 3D FC FF FF 20 B4 A3 00 00 4A 4A 4A 4A 4A 4A 4A 16 19 2E B4 A3 00 00 00 AB 94 00 00 30 00 00 00 00 00 00 1B 1A 63 00 00 00 00 00 00 35 D4 44 00 00 D6 44 00 00 00 00 10 00 A8 94 00 00 00 B1 94 00 00 16 19 00 AB 94 00 00 30 00 00 00 00 00 00 1B FC 62 00 00 00 00 00 00 35 D5 44 00 00 D6 44 00 00 00 00 10 00 A8 94 00 00 00 B1 94 00 00 16 0F 19 00 AB 94 00 00 09 00 B9 A2 00 00 00 01 B9 A2 00 00 10 00 A8 94 00 00 00 AC 94 00 00 0F 00 AA 94 00 00 1B B0 66 00 00 00 00 00 00 20 44 B8 00 00 01 9B 93 00 00 19 00 A7 94 00 00 09 00 DE F8 FF FF 00 01 DE F8 FF FF 19 00 A7 94 00 00 09 00 DD F8 FF FF 00 01 DD F8 FF FF 00 AB 94 00 00 01 E3 93 00 00 4A 00 A7 94 00 00 4A 4A 16 0F 19 00 AA 94 00 00 09 00 3D AF 00 00 00 01 3D AF 00 00 35 D0 44 00 00 D6 44 00 00 00 00 10 00 A8 94 00 00 00 B1 94 00 00 19 00 AA 94 00 00 31 00 00 00 00 00 00 1B 5C 02 00 00 00 00 00 00 35 D3 44 00 00 D6 44 00 00 00 00 10 00 A8 94 00 00 00 B1 94 00 00 4A 16 07 6C 03 72 00 AA 94 00 00 2A 16 0B 0B 0B 07 1D 04 9A 35 D3 0D 00 00 D5 0D 00 00 00 00 19 00 AB 94 00 00 09 00 C3 A2 00 00 00 01 C3 A2 00 00 2C 02 16 19 2E B5 46 00 00 19 00 AA 94 00 00 0A 00 12 33 00 00 00 1B EA 33 00 00 00 00 00 00 16 45 00 00 00 00 00 00 1B 56 61 00 00 00 00 00 00 35 32 0E 00 00 3D 0E 00 00 00 01 19 2E B4 A3 00 00 00 AB 94 00 00 09 00 79 A3 00 00 00 01 79 A3 00 00 4A 16 12 20 FC A8 00 00 14 00 00 00 00 00 00 1B 03 04 00 00 00 00 00 00 00 AA 94 00 00 4A 16 19 00 AA 94 00 00 0A 00 00 00 00 00 00 1B 41 6E 00 00 00 00 00 00 16 A5 00 A8 94 00 00 16 06 3A 01 04 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 53 00 00 00 02 00 02 00 1A 44 00 00 00 00 00 00 

Resulting view tokens:

(470/2E0) [06 3A 01]
	J(3/3)
	goto J0x13A

(473/2E3) [04 0B]
	R(2/2) -> N(1/1)
	return

(475/2E5) [0B]
	N(1/1)
	
(476/2E6) [0B]
	N(1/1)
	
<...>

(53C/3AC) [0B]
	N(1/1)
	
(53D/3AD) [0B]
	N(1/1)
	
// The structure is as follows:
// 	(MemoryPosition/StoragePosition) [Bytecodes]
// 		Token(MemorySize/MemoryPosition) -> ...
// 		Code

 

 

 

[Amineri]

If the memory size in the header is smaller than the actual memory size of the function then UE Explorer won't show the EOS token.

 

I've seen this crop up several times when my changes result in a memory size that is larger than the original function. I have to boost the function memory size in order to get UE Explorer to display to the EOS token. I generally do this by adding 256 memory bytes (0x100).

 

However, it's odd that it would be happening when you are replacing a bunch of code with null operators. I'll poke around and see if I notice anything.