New tool available that recalulates the jump offsets automatically

[Bertilsson]

Thank you for that advice. I will probably look into it sometime around the weekend when I expect to have version 1.0 ready.

 

Right now I'm trying out different layouts.

So far I think this is an improvement where less is more:

http://hem.bredband.net/bertrich/XCOM/Temp/RepairTool081.png

• The user isn't attacked with too much information at once
  • Old target is now represented via hovertext together with offset for each availalbe target location
  • Line numbers are removed
  • Target selectors only show the target address while not expanded due to ongoing selection
• The target selectors are now embedded inside the code where they are more intuitive
  • No need to think about line wrapping any more, since the target selectors are not depending of code line length.

I have also planned to include a new feature, basically a power selection, which means that whenever a target is changed, all later references pointing to the same old target will automagically be changed to the new target. If user don't like it all he has to do is go to the first power modified selector and change it back for all selectors below to revert back to original via the same mechanism.

Edited October 3, 2013 by Bertilsson

[Amineri]

The only comment I'd have is that the UI appears to missing any indentation to give the user feedback as to when the jumps are actually set up correctly.

 

For a basic switch/case statement as shown it's no big deal, but more complex nested if/else statements or conditionals nested inside switch/case for example might be a bit tougher to understand.

 

I'm not sure how difficult it would be to add such indentations, or what some of the special case rules would be, particularly like with things such as break; statements (which oddly enough UE Explorer correctly references as break statements in the object view, but displays as goto statements in the token view ...).

 

If you are able to add indentation it could be both a good thing and a bit of a problem. I'd suggest leaving the address at the front of the line untouched, and apply the indentation to the decompiled C-style code. This of course would require a larger box to handle the possibility of considerable indentations (with multiple nestings, for example).

 

Here is an exmaple of one of the more complex set of nested mix of conditionals and switch/case statements I've put together. It's in XGAction_Fire.SetChainedDistance . It re-defines how both grenade-type, rocket-type, and a few psionic abilities define their maximum range in free-aim mode. It's in the current Long War (has been since v1.9).

 

 

 

simulated function bool SetChainedDistance(XGTacticalGameCoreData.EAbility eInputAbilityType, optional out float fMinDistance)
{
    local TWeapon tempTWeapon;

    // End:0x4A0
    if(AbilityRequiresChainedDistance(eInputAbilityType))
    {
        fMinDistance = float(1);
        UnitX = 0;
        UnitY = 0;
        switch(eInputAbilityType)
        {
            // End:0x61
            case 22:
                UnitX = 85;
                // End:0xA3
                break;
            // End:0x75
            case 24:
                UnitX = 88;
                // End:0xA3
                break;
            // End:0x89
            case 23:
                UnitX = 86;
                // End:0xA3
                break;
            // End:0x9D
            case 71:
                UnitX = 99;
                // End:0xA3
                break;
            // End:0xFFFF
            default:
                // End:0xA3
                break;
        }
        // End:0x1CD
        if(UnitX != 0)
        {
            tempTWeapon = XComGameReplicationInfo(class'Engine'.static.GetCurrentWorldInfo().GRI).m_kGameCore.GetTWeapon(UnitX);
            // End:0x166
            if(false)
            {
                UnitY = (64 * (m_kUnit.GetOffense() - 65)) / 10;
            }
            UnitX = (tempTWeapon.iRange * 64) + UnitY;
            // End:0x1CD
            if(m_kUnit.HasBombardAbility())
            {
                UnitX *= 1.50;
            }
        }
        // End:0x22A
        if(UnitX == 0)
        {
            switch(eInputAbilityType)
            {
                // End:0x206
                case 73:
                    UnitX = 6400;
                    // End:0x22A
                    break;
                // End:0x20B
                case 76:
                // End:0x210
                case 74:
                // End:0x224
                case 35:
                    UnitX = 1728;
                // End:0xFFFF
                default:
                    // End:0x22A
                    break;
                }
        }
        // End:0x465
        if(UnitX == 0)
        {
            switch(eInputAbilityType)
            {
                // End:0x251
                case 25:
                // End:0x45F
                case 66:
                    // End:0x2D8
                    if(m_kUnit.GetInventory().HasItemOfType(19))
                    {
                        UnitX = 19;
                        // End:0x2D5
                        if(true)
                        {
                            UnitY = (64 * (m_kUnit.GetOffense() - 65)) / 4;
                        }
                    }
                    // End:0x2E4
                    else
                    {
                        UnitX = 7;
                    }
                    tempTWeapon = XComGameReplicationInfo(class'Engine'.static.GetCurrentWorldInfo().GRI).m_kGameCore.GetTWeapon(UnitX);
                    UnitX = (tempTWeapon.iRange * 64) + UnitY;
                    // End:0x3DE
                    if(m_kUnit.GetCharacter().HasUpgrade(29))
                    {
                        UnitX *= 1.340;
                    }
                    // End:0x45F
                    if(m_kUnit.m_iMovesActionsPerformed > 0)
                    {
                        // End:0x44F
                        if(m_kUnit.GetCharacter().HasUpgrade(16))
                        {
                            UnitX *= 0.750;
                        }
                        // End:0x45F
                        else
                        {
                            UnitX *= 0.50;
                    }
                }
                // End:0xFFFF
                default:
                    J0x465:
                    // End:0x465
                    break;
                }
        }
        // End:0x4A0
        if(UnitX != 0)
        {
            m_kCursor.m_fMaxChainedDistance = float(UnitX);
            return true;
        }
    }
    return false;
    //return ReturnValue; 
} 

 

 

 

As you can see it's a fairly complex mess. I actually indent my hex-code/decompiled-code files specifically to help me manage how I should be setting the jump offsets, as it's all too easy to get mixed up and end up with a weird logical error.

[Bertilsson]

Currently it correctly handles indentation (and brackets) for all if, case and foreach tokens, but it completely ignores any indentation which is the result directly or implicitly from goto-tokens.

 

The implementation is very rudimentary and basically just adds an indented end-bracket above and negatively adjusts indentations size for each non-goto jump-reference targeting every code line, by directly adding to the code line itself.

 

Start brackets and increased indentation is even less sophisticated as it simply adds a line break, current indentation and a start bracket directly to the end of any code line which holds a non-goto jump reference.

 

In order to handle indentation and brackets based on goto-data I would basically have to stop cheating (by directly adding to the code line) and create a proper indentation model where goto's can be anything from a simple non-indented goto-reference to being responsible for transforming itself into high level meta data affecting other sections of data.

 

I would also have to make changes to the target selectors to make them reflect what kind of surrounding effect the different alternatives will have (example a goto in the end of an if-section could result in a for loop, exit for, else or else if statement depending on which target is selected and the context of which the token exist (such as inside an if-statement inside a for-loop inside a case).

 

I guess, nothing of this would be very technically challenging but it would definitively take a significant lot more effort to implement than what I have spent creating the current model.

 

But I do agree that it would be nice to have more high level representation of the code directly in the tool, so it is on the to-do-list but very close to the end in a section of the list I may never end up implementing :smile:

 

Short term it is more likely that I will add some //comments for each line targeted by a goto jump reference to make it easier to spot errors and possibly some //comments after the goto selectors themselves when it can be simply determined what the goto represents.

Edited October 4, 2013 by Bertilsson

[Bertilsson]

The only comment I'd have is that the UI appears to missing any indentation to give the user feedback as to when the jumps are actually set up correctly.

 

For a basic switch/case statement as shown it's no big deal, but more complex nested if/else statements or conditionals nested inside switch/case for example might be a bit tougher to understand.

<cut>

As you can see it's a fairly complex mess. I actually indent my hex-code/decompiled-code files specifically to help me manage how I should be setting the jump offsets, as it's all too easy to get mixed up and end up with a weird logical error.

On second thought... Doesn't UE Explorer already do an excellent job for this, except that it masks jumps to invalid addresses?

 

The intended way to use the tool is:

1. Export function (bytes) from UE Explorer into a separate file for the function you want to edit
2. Mod the file using HxD or any other hex-editor
3. Import the modded file back into UE Explorer to get new view tokens
4. Copy view tokens into the repair tool
5. Copy byte code into the repair tool (if you only want to verify valid header and jump offsets you can actually skip this step)
6. Fix anything broken and adjust jump offsets to liking
7. Replace the code using HxD or any other hex-editor
8. Import the new code back into UE Explorer again

If you have already done the editing by hand you only have to do step 4 and nothing else.

 

If you have created a complicated else if/else statement, you can simply verify in the tool that all jump offsets are pointing to valid target (you will be informed about any identified issue) and then afterwards you verify in UE Explorer that the object code looks like intended...

 

The reason I recommend exporting/importing in UE Explorer instead just modding the .upk file directly is:

• You don't have to bother about file offset
• You can move code around using CTRL + X + V without worrying that you will break the entire .upk if you make a mistake
• You can quickly save and restore different versions
• You can use CTRL + A to quickly copy and replace the entire function

Edited October 5, 2013 by Bertilsson

[Bertilsson]

I've done some additional improvements to the tool.

• Color coded token-status
  • Broken (invalid) jump offset selectors are marked red and literately ask for HELP
  • Orginal valid jump offsets are marked green.
  • Tokens manually or automatically modified by the tool are marked yellow
• Status column has 4 alternatives: Broken, Original (valid), Manual (user selected) and Automatic to make it possible for color blind to keep track and easier to find potentially unwanted automatic changes.
• A big status indicator above the code informing if any broken jump tokens exist
• Offset inheritance
  • If later (downwards) tokens has a valid target available at the same offset from original target as the user modified, then that target is automatically selected for the later tokens as well.
  • This is pretty much the core principle that the old tool was based on, with the difference that it must be manually triggered by the user and that it is very transparent and easy to spot and reverse any unwanted changes.
• Censuring of 0B tokens
  • As far as I know there is absolutely no scenarios where you would need to target one of those instead of the target line immediately following them
  • Not having them clutter up the code representation is much nicer than having them there for authenticity feeling.

 

Features left to implement before v1.0:

• Some basic validation rules for the text boxes where byte code and view tokens are inserted
  • Currently pretty much anything is accepted as valid input and changes are applied to thin air if no byte code is available to modify where it is expected to be.
    • I kind of like it since it makes debugging a lot faster but it is not very pretty
• Warning message about or attempt to handle relative offsets inside complex foreach tokens
• Major code cleanup
  • There is LOTS of room for improvement in this area but not really prioritized as long as it does what I want it to do and it is still (barely) possible to read by myself :smile:

Feature that I am still undecided about if it will be implemented in a later version:

• Logical presentation of goto tokens shown in the object code as else and loops, etc

Feedback that I am primarily interested in right now:

Does the tool behave irrational, incorrectly or unexpectedly in any real life scenarios?

As far as testing goes it seems to work as planned but real life has a tendency to serve surprises...

Edited October 6, 2013 by Bertilsson

[Bertilsson]

I've done some additional thinking regarding the "to else or not to else" question and the more I think about it would actually be nice to see directly in the tool what is the else-section.

 

From my understanding an else-statement in UE Explorer is nothing more than:

A "forward goto-token" located exactly 3 memorybytes before the (if not) target of an if-token.

The range/length of the else-section is defined by anything between the goto-token and the target of the goto-token.

 

Else if-statements aren't explicitly modeled, instead they are simply if's inside an else-section.

10. if condition  //otherwise goto 40
    {
20.   bla bla bla
    }
30. else //goto 80
    {
40.   if condition  //otherwise goto 70
      {
50.     bla bla bla
      }
60.      else //goto 80
      {
70.     bla bla bla
      } 
    }
80. bla bla bla

So basically all I would need to do is to add the word else before any goto token which fits the critera of going forward and being at the very end of an if-section, adjust the end-bracket of the if to show above the goto instead of after where it really belongs and add indentation for the else-sections...

 

Actually does not seem too difficult and will most likely be implemented the next time I find some time to work on the tool. :smile:

 

I think I will actually add the words "otherwise goto" between the if condition and the target selector of if statements while I'm at it to make it more consistent and easy to understand.

 

Another feature I'm considering is to add a remove-button for each code-line which will simply place the storage bytes as 0B tokens after the 53 token and adjust the affected jump offsets accordingly.

 

Possibly that feature could be expanded to allow moving code and injecting code similarly and ultimately allow for automatic adjustments of the relative offsets of complex foreach statements as well. But that is not likely to happen very soon.

[Amineri]

That's exactly how if/else constructions are built. The else is in a sense just an unconditional jump. Since this is represented in hex via 06 ## ## it is always 3 file/memory bytes (same in this case) long. This is why the 07 jump if not address is 3 bytes offset from the unconditional jump location.

 

One interesting effect of the way that nest if/else statements are translated into the hex code is that this can result in a series of "cascading gotos" in certain conditions:

10. if(condition1) // otherwise goto 70
    {
20.    if(condition2) // otherwise goto 50
       {
30.       bla bla bla
       }
40.    else // goto 60
       {
50.       bla bla bla
       }
    }
60. else // goto 80
    {
70.    bla bla bla
    }
80. continuation of code

The goto cascade in this example is that line 40 jumps to line 60 which immediately jumps to line 80. The hex code would function identically if line 40 were to jump directly to line 80, but this would not reverse compile correctly into the series of nested if/else statements.

 

With 3 nested if/else statements with the if/else progressively nested in the 'if' portion, you would see a series of 3 gotos jumping to each other.

 

Just one of the funny things to keep in mind as you are building the logic for replacing goto's with else statements.

 

----------------

 

On a side note, I find replacing 07 ## ## <condition> statements by simply changing the 07 to a 06 (and changing the conditional jump to an unconditional jump) to be a very effective debugging technique.

 

Changing the 07 to 06 forces the code to skip the first part of the conditional, thus quickly determining if that branch is resulting in a CTD, and does so in a way that won't mess up file/memory byte counts. It also will skip the evaluation of the conditional statement.

 

In order to force the first branch of an if/else conditional to execute simply change the 07 ## ## to 0B 0B 0B. The conditional will still execute, but most conditional boolean evaluations have no side-effects of running, and so the result is simply discarded. The else portion will be skipped by the 06 ## ## unconditional jump at the end of the section.

[Amineri]

Oh, and I thought of another condition under which an if statement will jump to a location immediately following a goto statement : while loops.

 

A while loop is constructed like so:

10. I = 0; // iterator intiation

20. if(I < value) // goto 53
    {
30.    bla bla bla

40.    I++; // iterator advance

50.    goto 20; // while loop
    }
53. continuation of code

In this case the if statement ALSO jumps immediately after the goto statement.

 

The way to tell whether the goto is a while loop or an else statement is really based on whether the goto is going forward or backward.

 

1) If the else jumps forward than it is constructing an if/else construct

2) If the else jumps backward to an if statement whose address lies at +3 bytes to the else statment location then it is a while loop construct

3) If the else jumps backward otherwise then it is spagetti code :wink:

 

UE Explorer doesn't actually reverse compile these into standard C-style while loop source code but instead leaves the goto statement in place but with a comment indicating that it is part of a while loop. This is a little inconsistent as UE Explorer does remove the iterator next and iterator pop tokens from the decompiled source code.

Edited October 9, 2013 by Amineri

[Bertilsson]

Thank you for the feedback.

 

The while loop seems trivial to implement. Just check if the if-tokens targetline-1 is a goto-token pointing back to the if-token and if that is the case, change the word if to while and optionally make a comment at the end of the goto-line.

Else check if it is an "else-token" and flag it for Houdini-move to the other side of the if-ending-bracket and rename it accordingly.

If both above checks fail the tool should just treat it as an independent goto-token.

 

Regarding cascading gotos I don't see any simple logic that could be implemented to deal with it, but I'll keep that kind of scenarios in mind and maybe it will come to me eventually.

[Bertilsson]

I started implementing a few basic rules for the while and else-logic when I discovered that this one was actually a little trickier than first expected:

1) If the else jumps forward than it is constructing an if/else construct

 

If the line before the target of an if-token is a goto-token that is targeting something past itself. Then it is either an else-statement or possibly an explicit break of some kind which I would be better of leaving as a plain goto and at the very least not treat as an else-token.

 

Example in DamageTypeHitEffectContainer.GetHitEffectsTemplateForDamageType

simulated function XComPawnHitEffect GetHitEffectsTemplateForDamageType(class<DamageType> DamageType, optional bool bExactClass)
{
  local int FoundIndex;

  bExactClass = false;
  FoundIndex = 0;
  J0x10:
  // End:0xE8 [Loop If]
  if(FoundIndex < DamageTypeToHitEffectMap.Length)
  {
    // End:0x71
    if(bExactClass && DamageTypeToHitEffectMap[FoundIndex].DamageTypeClass == DamageType)
    {
      // [Explicit Break]
      goto J0xE8;
    }
    // End:0xDA
    else
    {
      // End:0xDA
      if(!bExactClass && DamageTypeToHitEffectMap[FoundIndex].DamageTypeClass.IsA(DamageType.Name))
      {
        // [Explicit Break]
        goto J0xE8;  <<-- this one is falsely interpreted as beginning of an else-statement
      }
    }
    ++ FoundIndex;
    J0xE8:
    // [Loop Continue]
    goto J0x10;
  }
  // End:0x127
  if(FoundIndex < DamageTypeToHitEffectMap.Length)
  {
    return DamageTypeToHitEffectMap[FoundIndex].HitEffect;
  }
  return none;
  //return ReturnValue;  

One way to deal with the specific example above would be to implement a rule that ignores else-candidates if they are broken (targeting non-existing address) or targeting another goto-token (or possibly a foreach loop terminator?)

 

I guess it could also be more generous and only ignore else-candidates if they are broken or targeting a loop repeater/terminator.

 

Any alternative suggestions to the above? Would either of them solve the problem in general? Or would I just solve this particular example and run into problems in other scenarios?

Edited October 10, 2013 by Bertilsson