R&D - New Modding Tools

[wghost81]

The main thing I haven't internalized (but is provided in wghost's link) is just how the linking of local, return value and parameters is done. I'm sure I can figure it out, it's just that I haven't done it yet. At some point I had to stop working on the tool and start actually using the tool to work on the Long War EW mod :smile:

It is "linked" (or you may say "marked") by ReturnParm flag in PropertyFlags. Params (regular, optional, output) are also marked by PropertyFlags. So you need to construct the chain of function children and "mark" all child with corresponding flag.

 

I haven't included adding new objects into PatchUPK, as it is a big change and can not be handled safely with patching style. I was planning on making it as a separate modding tool, but since I had no need for this yet and no other modders have expressed need in such tool, I decided to return to actual modding for a change. :smile:

 

Regarding the adding of new classes, in a PM from amelocik he made the suggestion that for creating new classes we could use the existing Unreal Development Kit to develop stand-alone unreal package files, and then add import references to the new upk to the existing XComGame.upk and XComStrategyGame.upk. It might be simpler to add new classes that way then going through all of the steps needed to add enough objects to create a new class, default class, class variables, and class methods into an already cooked upk.

Good suggestion! In my current experiments I use a copy-paste approach. :smile: I just find an object, similar to what I need, copy it and make some modifications. I see one problem with UDK, though: it won't be able to work with existing XCOM objects. But yes, it could help create purely separate classes.

[XMarksTheSpot]

Regarding the adding of new classes, in a PM from amelocik he made the suggestion that for creating new classes we could use the existing Unreal Development Kit to develop stand-alone unreal package files, and then add import references to the new upk to the existing XComGame.upk and XComStrategyGame.upk. It might be simpler to add new classes that way then going through all of the steps needed to add enough objects to create a new class, default class, class variables, and class methods into an already cooked upk.

An interesting point, indeed. Especially so if it extends to importing new art assets, that I would like to see :)

 

X.

[renfang]

I've been trying to use upk modder 0.76 in order to apply this mod to show soldier mobility stats here:

 

http://forums.nexusmods.com/index.php?/topic/1111907-display-soldiers-xp-and-mobility-in-character-card-view/page-2

 

I've had no luck figuring out what i'm supposed to be doing in order to apply the upk_mod file, can anyone give me a step-by-step procedure?

[Amineri]

I've been trying to use upk modder 0.76 in order to apply this mod to show soldier mobility stats here:

 

http://forums.nexusmods.com/index.php?/topic/1111907-display-soldiers-xp-and-mobility-in-character-card-view/page-2

 

I've had no luck figuring out what i'm supposed to be doing in order to apply the upk_mod file, can anyone give me a step-by-step procedure?

 

Sure can!

 

Step 1) Decompress your upk files using Gildor's Decompress tool -- UPKmodder doesn't do anything or know anything about this (alternately ToolBoks will auto-decompress the upks and clean up the CookedPCConsole when launched)

Step 2) Open UPKmodder, create a new Project

Step 3) Create a new file within the project, paste the contents linked above into the new file

Step 4) Repeat step 3 for each file in the link

Step 5) Set up the target upks for the project using the control in the lower left (click on the folder icon to bring up a chooser) -- select the appropriate upk

Step 6) Perform step 5 for each type of upk (XComStrategyGame.upk and XComGame.upk) - the project stores the links from the UPKFILE= line in the file and will automatically set the targets after that

Step 7) Test the status of the upk_mod files, either by right-clicking the project and selecting "test status" or performing it for each file individually (green = BEFORE, blue = AFTER, red = ERROR)

Step 8) All files should show as green (unless you've already modded the upk in same way)

Step 9) Open each file and select the "Apply" button -- if successful the color should change to blue

Step 10) Launch the game

 

I realize that there are quite a few steps, even for installing a relatively simple 4-part mod, but the tool isn't optimized around end-user installs or small mods.

 

JL and I are using UPKmodder for Long War EW development (all AS changes are also rolled into the format), and currently it's managing 615 upk_mod files that go into the current Long War EW alpha.

[dubiousintent]

@Amineri: Added these instructions to a new 'Application' section of the UPKModder Feature page. Re-worded slightly to be more general, so please review for accuracy.

 

-Dubious-

[renfang]

Thanks very much for the help, my problem was the decompression of the .upk files, I was trying to load compressed .upk files.

[wghost81]

2Amineri

 

I've modified PatchUPK mod file format to be able to quick-convert (manually) from UPKModder format. But I have some questions.

 

Here is an example of UPKModder base syntax:

[BEFORE_HEX]
[HEADER]
header bytes
[/HEADER]
[CODE]
code bytes
[/CODE]
[/BEFORE_HEX]

Does UPKModder searches for "code bytes" data inside function script only? Can you specify "header bytes" and then specify some "code bytes" from the middle of script? When you expand a function, do you add "null ops" (or zeros, or something else) before or after 53 token? Do you search for BEFORE_HEX prior to function expanding procedure or after that? Edited February 18, 2014 by wghost81

[Amineri]

 

I've modified PatchUPK mod file format to be able to quick-convert (manually) from UPKModder format. But I have some questions.

 

Excellent. I'll be happy to answer any and all questions.

 

 

Does UPKModder searches for "code bytes" data inside function script only?

 

 

 

The [ CODE ] ... [ /CODE ] delimiters are only used to indicate what UPKmodder should attempt to perform limited de-compilation on, which provides the highlighting of various hex elements. It also allows it to identify object/name references to be used by the reference updater tool. Further, correct line memory position computation requires knowledge of object references (each object references adds +4 memory bytes).

 

It is perfectly valid to have replacements without CODE blocks, or even to mix hex CODE and non-CODE blocks within a BEFORE or AFTER block. However this isn't very patch-safe, since the references can't be identified, and memory positions of lines can't be computed.

 

Can you specify "header bytes" and then specify some "code bytes" from the middle of script?

 

 

 

Not within the same BEFORE or AFTER block. All of the hex bytes within a single block (whether wrapped in HEADER, CODE, or "raw") should be one contiguous block of hex within the upk.

 

If you want to separately replace the header and a chunk of hex within the function body you can do so using multiple BEFORE / AFTER blocks.

 

 

When you expand a function, do you add "null ops" (or zeros, or something else) before or after 53 token?

 

 

 

Function expansion in UPKmodder works a little differently than in PatchUPK, as the resize and replace take place as a single step, so no null ops are ever automatically added.

 

Here's basically what happens:

1) UPKmodder checks BEFORE and AFTER size, compares with the RESIZE parameter and confirms that they match (i.e. BEFORE + RESIZE = AFTER)

2) A .bak copy of the upk is made

3) The data from start to prior to first byte of FIND block is quick-written from the .bak to the new upk

4) The REPLACE hex block is written to the new upk

5) The data from end of FIND block is quick-written from the .back to the new upk

6) All object table entries subsequent to the modified one are adjusted by RESIZE

 

Because the REPLACE block is a different size than the FIND block, the upk ends up a different size. This is somewhat different than how PatchUPK does it, with a 2 step resize, then replace operation.

 

When applying a change FIND=BEFORE and REPLACE=AFTER, and when reverting a change FIND=AFTER and REPLACE=BEFORE

 

Do you search for BEFORE_HEX prior to function expanding procedure or after that?

 

 

 

Since it happens as a single operation, there is a bunch of error checking (as much as I could add) prior to any form of resizing.

 

Error checks include:

• Object being resized is an export table object (import objects cannot be resized)
• Reference FUNCTION/OBJECT was found in the upk
• There is only a single BEFORE and a single AFTER block
• Verifies that BEFORE block size + RESIZE amount == AFTER block size
• It finds position within the file by searching for the FIND block (either BEFORE or AFTER if apply or revert, respectively), so if the hex isn't found the operation aborts

I've functionally broken the resizing into 2 steps internally to UPKmodder, but it isn't possible to execute them separately:

Step 1) Copy upk in 3 chunks (old upk beginning hex + REPLACE hex + old upk ending hex) -- this temporarily results in an invalid upk

Step 2) Loop through UPK object list, adjusting file position of every entry after the current one, and modifying the size of the current object -- this repairs the upk

 

It's perfectly possible for the user to add additional 0B null ops to an resize function, and to add them either before or after the 53 end-of-script token. I typically add them just prior to the EOS token, but either is perfectly valid.

 

 

--------------------

 

I'll add that the [ HEADER ] block currently doesn't have any real functionality attached to it. The reason I included it was to allow for future functionality such as automatic repairing of the memory/file sizes. However currently we are still manually updating these.

[Amineri]

I should add that the rather generic resizing functionality (which is not particularly set up for functions) is what allowed for easy resizing of enums.

 

Technically the resizing action knows nothing at all about function formatting, either the header size, header composition, footer size, or footer composition. It simply replaces one hex chunk with another (differently sized) hex chunk within the specified object, then performs necessary corrections to the object list in the upk header.

 

This means that it's up to the modder to correctly adjust the header memory/file sizes, or to specify any additional bytes in the footer to be changed.

 

It's perfectly possible to set up a upk_mod file to resize a function and include all of the 48 bytes of header and 15 bytes of footer information. In practice on Long War we don't do this because I haven't added parsing of the header or footer, and it includes references that can change from patch-to-patch. So currently we include only the final two ints of the header and none of the footer. That way every reference is parsable (if within CODE blocks, of course), which makes it simpler to update.

 

There are a few non-patch-safe operations we're doing, primarily involving resizing of static arrays, as I haven't added the capability to parse objects representing variables.

[wghost81]

Thanks for the answers, Amineri.

 

I've modified EXPAND_FUNCTION key in PatchUPK to work like this:

EXPAND_FUNCTION=UIStrategyComponent_SoldierStats.UpdateData:1254
or
EXPAND_FUNCTION=UIStrategyComponent_SoldierStats.UpdateData:+0xEE

First form requires absolute object size and second requires resize amount similar to RESIZE key in UPKModder.

 

Since PatchUPK uses BEFORE_HEX to determine offset only and performs resize by moving an object, converting a whole function script from UPKModder format is easy. But, if I understood correctly, UPKModder can take, say 100 bytes in the middle of an object and resize that part to, say, 200 bytes in-place. This type of operation will not be easy to convert, but I will think of something. :smile:

 

PatchUPK will now recognize C-style comments too.

 

I was able to manually convert "Display XP and Mobility" mod in a matter of seconds. But still, we'll need to make some tool for automatic conversion.

Edited February 18, 2014 by wghost81