Jump to content

R&D - New Modding Tools

Amineri

By Amineri
in Enemy Unknown

 Share

Recommended Posts

johnnylump Experienced

johnnylump

Posted
Posted (edited)
  Quote

 

And when it comes to XCom EU. The 4 first patches... Were any of them friendly in nature where references did not change or were they all resulting in new references? (I don't know, but am sort of curious if the advantage of search & replace strategy really is relevant for this specific game) :smile:

 

To be honest, we'd barely cracked modding the upk beyond simple value changes when patch 4 came out -- nearly all the work figuring out how functions worked and rewriting them came after patch 4. I do recall some reference changes (not because I was cross-utilizing variables like we do now, but because my search strings no longer worked). In other words, the impact of patches is now vastly greater because we're doing so much more complex work.

Edited by johnnylump
Link to comment
Share on other sites
  • Replies 211
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Amineri Veteran

Amineri

Posted
  • Author
Posted

I've been working on a new modfile definition, borrowing heavily from those that preceded me but focussing primarily on "what I want".

 

It's not nearly as deep as a full XML implementation, looking more like a ToolBoks Custom Mod file.

 

Allowed syntax:

  • // and /* */ style commenting
  • UPKFILE=<name> declarations (needs to be extended to handle before/after upk files for patch updating)
  • FUNCTION=<name> declarations (used to limit scope of search/replace operations)
  • [bEFORE_HEX] and [/bEFORE_HEX] delimiters wrap hex being replaced
  • [AFTER_HEX] and [/AFTER_HEX] delimiters wrap new hex
  • ||<string>|| statements within hex delimiters allow for using named references
  • {{ ## ## ## ## }} statements within hex delimiters allow tagging of hex references
  • [[ ## ## ## ## ]] statements within hex delimiters allow tagging of external hex references -- I'm not quite sure where these are stored

Here is a small snippet of an example/test modfile I'm working on:

// sample comment

/* more 
sample
comments */

UPKFILE=XComGame.upk
FUNCTION=AddStatModifiers.XGUnit

[BEFORE_HEX] // hex from EU patch 4
//final 2 words of header (function memory/file size)
A6 02 00 00 DE 01 00 00 

// body
//kAbility = none
49 02 00 2A 15  

//iStat = 0
0F ||iStat.AddStatModifiers.XGUnit|| 25 // example of reference-naming

//if(iStat < 19)
07 56 00 96 00 {{ 39 B7 00 00 }} 2C 13 16 // example of reference-marking

As an exercise I went through and created a more realistic example using the Long War First Aid Kit hex change (one of the functions, anyhow).

 

 

  Reveal hidden contents

 

 

The display looks a lot prettier for me in Notepad++ as I created a custom language type that allows for coloring based on the various tag types.

 

Before/after upk files for performing updating would be configured separately, not within the modfile itself.

 

What I like about this format is that it allows for UE Explorer Tokens View-style breakdown of the hex line-by-line. References can be tagged and kept as hex (which seems faster to me) or identified as strings.

 

This file could either be:

1) Converted into a patch-specific before/after set of replacement hex

2) Read and interpreted and directly applied to a upk

 

In terms of workflow the problem with directly applying such changes is correcting jump offsets and function header memory size.

 

 

In this example I went through and marked all of the references by hand, which is a bit cumbersome. I didn't use XML since I'm doing my test-coding in Java (which I'm not very familiar with), and handling the overhead of XML was more than I wanted to mess with :)

 

Open to thoughts/suggestions!

Link to comment
Share on other sites
wghost81 Mentor

wghost81

Posted
Posted (edited)
  Quote

I view Game().GetDifficulty() like this:

 

Probably19 + ReferenceToGame() + SomeMoreOrLessFixedLengthJunkThatAmineriTendsToEndUpHelpingMeWithAfterIFailAFewTimes + ReferenceToGetDifficulty() + 16

This junk is local ReturnValue reference. Basically, what you wrote is a manual code compilation. Parsing variables and function calls only is easier than full code compilation, but it is still a very complicated task. But, may be it's just my lack of skill. :smile:

 

Amineri, the biggest problem I had with ToolBoks custom mod file format was necessity to copy/past the existing binary code to text document. First, some functions are quite big and when you have about 10 big functions in one text file it's a mess. Second, you need to keep track of several separate files, like your working copy, exported binary function etc. I don't think anyone starts to write a mod using ToolBoks custom mod feature, one usually starts with modifying hex code either in upk or in exported function. And when mod is ready and tested a question about how to distribute it arises. In this case, IMHO, best approach would be to add BINARYFILE field with link to external binary function/state/property file:

OFFSET=value
BINARYFILE=Path/To/BinaryFile
It will simplify matters A LOT, at lest for me. So right now I'm working on tool which will take a list of files like

XGStrategyAI.ProcessPodTypes.Function
XGAIBehavior.ExecutingAI.State
XGStrategyAI.GetAltWeapon.Namelist
find their locations by name and apply changes from binary files.

 

PS The best tool ever should be flexible and should allow everything we just wrote. :smile:

Edited by wghost81
Link to comment
Share on other sites
Amineri Veteran

Amineri

Posted
  • Author
Posted

Well, here's how my current workflow goes in terms of creating a mod:

 

  1. Design -- figuring out which functions have to be changed and functionally what the new code will do. For some mods (like the alien base/retaking countries mod) this can be fairly complex
  2. Start with vanilla decompiled code
  3. Write target code (as unrealscript, not hex)
  4. Extract unrealscript-to-hex line-by-line correspondence using UE Explorer Token view
  5. Copy hex code to create line-by-line target code. Manually compile unrealscript to hex as needed. Use placeholder jump offsets and function header memory size.
  6. Consolidate to before/after hex blocks and insert into upk
  7. Use UE Explorer and Bertilsson's jump offset tool to compute corrected jump offsets
  8. End state is a before/after hex replacement block for the function
  9. Run code and debug

And here are the actual excerpts from my notes when I was modding the OnCodeCracked function for the multiple alien bases:

1) Design:

 

  Reveal hidden contents

 

 

2) Original code:

 

  Reveal hidden contents

 

 

3) Target code:

 

  Reveal hidden contents

 

 

4) Original unrealscript/hex:

 

  Reveal hidden contents

 

 

5) New unrealscript/hex

 

  Reveal hidden contents

 

 

6) Consolidated before/after hex blocks:

 

  Reveal hidden contents

 

 

7, 8 ) When I wrote this function I wasn't using Bertilsson's tool yet and iterated the code from steps 5 and 6 until the jump offsets were correct.

 

9) Debugging is typically done by making small temporary changes to narrow down where a fault is happening. Here are some examples:

 

  Reveal hidden contents

 

 

The complete ideal would be to have a fully automated compiler and be able to just write the target unrealscript (STEP 3). The compiler would convert it to hex code.

 

Short of that step 5 in my process is the first point at which I have new hex code, and still involves manually copying/compiling the unrealscript to hex. The tool I'm working to would allow taking that step and directly inserting it into the upk, or at least consolidating it, saving a few steps.

Link to comment
Share on other sites
wghost81 Mentor

wghost81

Posted
Posted

Steps 1-5 in my case are almost the same, but I found it inefficient to consolidate the code into one string. Initial script seldom works as expected :smile: and when I change the code/binary-code I need to change consolidated version as well. So my "compiled" code usually looks like this:

0F                                    let
00 EB 43 00 00                        iNumPods
FB                                    Clamp
92                                    sum
2C 02                                 int const byte (2)
00 EF 43 00 00                        iMissionDiff
16                                    end sum
2C 02                                 int const byte (2)
2C 04                                 int const byte (4)
16                                    end Clamp
I use UE Explorer to export binary function, HxD to fill function's body with 0B, then block-select a code without text and paste it into function with HxD. Then I use UE Explorer to import new function into upk. Speeds up process for me a lot.

 

Things after step 5 could use a little automation. I worked with Fallout 3 scripts before compiler came out and I wrote a simple tool which converted text files like this one to binary files. It helped me in copy/paste steps. :smile: With information about upk format and text files like this it could be relatively easy to patch upk with one tool and without additional steps. Manual re-calculation of jump offsets still stays, of course.

Link to comment
Share on other sites
Amineri Veteran

Amineri

Posted
  • Author
Posted

Not arguing that my workflow couldn't stand some improvement, not at all ^_^.

 

A lot of it was developed back before UE Explorer had the token view or could replace hex in the upk. Also before Bertilsson's jump offset repair tool.

 

My original workflow had step 4 taking much longer, as I had to copy out the function's hex as a single block and manually break it into lines. That one update of UE Explorer drastically shortened that step of the process. The reason I do step 4 generally is because the replacement function uses much of the same code as the original. Sometimes that isn't the case, such as when doing a complete rewrite of XGUnit.DebugAnims into a helper function to set leader/navigator stats/perks. In those cases I'll skip step 4.

 

The reason for consolidating into hex string blocks is that's basically how I "distribute". For a mod like the multiple alien bases/retaking countries I end up with a single file holding all of the hex changes. For patch 4 EU this was :

 

  Reveal hidden contents

 

 

This has 27 separate hex changes required, and also includes related localization changes.

My original mods I didn't have such a file but later I started creating them to help me keep track of all of the changes.

Link to comment
Share on other sites
Amineri Veteran

Amineri

Posted
  • Author
Posted

So I've got a Java tool that can now read and parse two upk files, as well as open and parse a modfile in a format like:

 

  Reveal hidden contents

 

 

The above hex references are all for EU patch 4.

 

The output of my (still work-in-progress) tool is :

 

  Reveal hidden contents

 

 

Basically creating before/after replacement hex blocks with all reference-related tags/formatting stripped out. I've verified that the above search and replace is valid in patch 5 EU.

 

At this point I'm wondering which is the better alternative:

  1. Use the more verbose form as the release form of a mod
  2. Use this purely as a mod development tool and release XCom patch-specific hex changes
Link to comment
Share on other sites
dubiousintent Grand Master

dubiousintent

Posted
Posted (edited)

As long as the mod installer can parse the verbose format, I would think that to be preferable. Primarily for those who want to tweak something: the verbose format makes each hex byte understandable in context.

 

But perhaps releasing another tool to strip the verbose format into pure hex would be feasible, assuming there is demand due to mod patch size or author wanting to discourage tweaking?

 

-Dubious-

Edited by dubiousintent
Link to comment
Share on other sites
Amineri Veteran

Amineri

Posted
  • Author
Posted

For my next mutant version of my proto-tool I've got a version that will automatically update tagged references with appropriate tagged string-names.

 

For example the my sample modfile has been turned into:

 

  Reveal hidden contents

 

 

Notice that there are some seemingly inconsistent cases such as the line:

// XComUnitPawn(m_kPawn).ApplyBloodCall()
19 2E ||XComUnitPawn|| 01 ||m_kPawn.XGUnitNativeBase|| 0A 00 00 00 00 00 00 1B ||ApplyBloodCall|| 00 00 00 00 16
  • XComUnitPawn is a singleton because it is an object reference to a class, so has no owner
  • m_kPawn is owned by XGUnitNativeBase, which is the parent of XGUnit
  • ApplyBloodCall needs no owner reference because it is a virtual function reference

All virtual functions with the same stringname have the same 4-byte hex reference, even if they are in different classes. I totally didn't realize this until working on this project, but have confirmed that it is so.

 

 

--------------------------

 

I think the next step I'm going to work on is trying to automate the marking of the references. If I use the line-by-line hex code together with the extended token view info from UE Explorer it might be possible.

Link to comment
Share on other sites
 Share
Go to topic listing

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...