R&D - New Modding Tools

[wghost81]

Yes, repeat-ability is good.

 

Create object and change type are different operations, so separating them is good too. But, IMO, CreateObject should work silently only if object is either not yet exists at all or is an absolute match to specified object. In any other case operation should fail and output an error. Same with SetType.

 

Command line tools and install scripts are great for distribution, but for actual modding process, when things tend to change, GUI application with ability to see and directly edit objects may be better. Although, it is still a matter of preference. :smile: I, for example, am perfectly happy with command line tools. :smile: I did had a passing though about creating some GUI UPK editor... but problem is - I am terrible with GUIs and an attempt to make one will cost me a LOT of time. :smile: Another problem is that seemingly I'm only C++ programmer here, so my C++ libraries are practically useless for the others, who code in Java.

[Amineri]

I approve of separating object and change. In general I'm more of a fan of error-checking and reporting than trying to have the tool "do the right thing" despite any other intentions I might have ;) Well over half of the SLOCs in my utility classes for manipulating upks are dedicated to error-checking, which is paired with (as best I can) detailed log messages about the nature of any errors.

 

I also agree about command line tools being better for distribution. The only possible exception is for "power modders" who want to install and manage multiple mods from different sources. In that case a GUI interface for a Mod Manager can be useful.

 

 

 

I did had a passing though about creating some GUI UPK editor... but problem is - I am terrible with GUIs and an attempt to make one will cost me a LOT of time. :smile: Another problem is that seemingly I'm only C++ programmer here, so my C++ libraries are practically useless for the others, who code in Java.

 

My first attempt at a command-line Reference update utility felt nigh-unusable to me, which prompted me to explore creating the GUI. I'm also pretty terrible with GUIs, but fortunately XMTS has been collaborating and making it all possible.

 

I don't think you're the only C++ programmer about, not by far. I'm probably slightly more proficient in C++ than Java (although that may be changing...). The main 2 reasons I picked Java were:

1) The SDK and nice IDEs were free and easy to install

2) I had a notion of cross-compatibility for OS X / Linux / etc users.

 

Then it turned out that XMTS works a fair bit with Java UI development, which pretty much set the path. Despite my never having programmed in Java prior to starting this little escapade.

 

It is unfortunate that we can't share your C++ libraries (or if it is possible I don't know how).

 

----------

 

As for repeat-ability...

 

For UPKmodder I'm not sure that it will be an issue. Attempting to install a modfile when it is already installed will simply fail. Recently I've been focussed on automating and displaying the install status of mods. Since the modfiles contain both original and modded hex, it is pretty straightforward to test which version (or neither) is installed in the target upk.

 

My current prototype uses small colored dot overlays in the project manager to convey the install status (or error state) to the user.

 

For straight-up distribution installs this isn't needed, but I think mostly renders the repeatability a non-issue ?

 

---------------

 

After that's in, my next step is going to be adding the option to directly specify the relative offset within the object, as that seems to be a pretty popular modding process (even thought it isn't mine). Plan is to still check to see if the FIND hex is present, in which case an warning dialogue will display which the user can override.

 

We use the same process with GUID checks when doing Reference value updates or installing. If they mismatch a warning dialogue pops up which can be overridden.

[Bertilsson]

As for repeat-ability...

 

For UPKmodder I'm not sure that it will be an issue. Attempting to install a modfile when it is already installed will simply fail.

That is sort of an issue by itself when it comes to releasing an update of a mod.

 

I don't think it would be very popular to ask players to verify steam cache between installing beta 1 and beta 2 of a mod... Or to make it mandatory to install beta 2 addon and beta 3 addon (in that order) before installing beta 4 addon when going from beta 1 to beta 4. Not to mention the effort involved in actually maintaining such version chains.

[Amineri]

For those following it, there's another incremental release of UPKmodder, v0.75. Here's the changelog:

 

 

 

v0.75 (r126)

Fixed error in FRand operator definition

Improved speed of search algorithm to better handle searches in large objects

Fixed bug preventing parsing of mixed hex + reference names

Added "Test Status" command as context-menu action in Project Pane. Tests all subfolder/files.

"Test Status" results displayed as icon overlays in both project pane and tab.

GREEN = BEFORE hex present

BLUE = AFTER hex present

GREY = No upk designated

YELLOW= Mixture of BEFORE, AFTER, and No upk

RED = Error

Hooked up progress bar to display progress when testing status of large projects.

Added confirmation dialogue for unsaved files when closing application.

 

 

 

The primary new feature is the capability to test the install status of all files in a single project.

 

Souce and application can be downloaded at http://code.google.com/p/upk-modder

[Amineri]

 

Amineri, on 30 Dec 2013 - 7:08 PM, said:

http://forums.nexusmods.com/public/style_images/underground/snapback.png

As for repeat-ability...

 

For UPKmodder I'm not sure that it will be an issue. Attempting to install a modfile when it is already installed will simply fail.

That is sort of an issue by itself when it comes to releasing an update of a mod.

 

I don't think it would be very popular to ask players to verify steam cache between installing beta 1 and beta 2 of a mod... Or to make it mandatory to install beta 2 addon and beta 3 addon (in that order) before installing beta 4 addon when going from beta 1 to beta 4. Not to mention the effort involved in actually maintaining such version chains.

 

I agree that for end-user installation of mods you want to be able to simply update.

 

However my primary design goal for UPKmodder isn't to enable end-user installations. At the time I started the project there were no less than three active installers (bokauk's ToolBoks, Ubjerjumper's XComModHelper, and wghost's UPKPatch), so my focus has been on building a tool to help create mods.

[Amineri]

Small update to UPKmodder v0.76 :

 

 

v0.76 (r128)

Change line number display into Memory Position display

Added new action "Generic Object Table Entry" change, and test

Corrected some errors/warnings when parsing modfile header values

 

JL and I have been using this enough now that I'd consider it to be pretty safe and stable for general modding use.

 

Updates to UPKmodder will be a bit slower, as I've shifted into more modding instead of development of modding tools.

[Amineri]

It's been a while since I've posted anything up on this, but that's because the tool has been stable enough that JL and I are using it quite a bit in the Long War EW development. We're currently closing in on 500 function/variable modifications, and this tool has been very helpful in making it happen.

 

We've been making quite noticable usage of the resizing capability, and it appears to be quite stable. I haven't counted, but I'm sure we have well over 50 function that have been increased in size, which is allowing us to do a lot of things we wouldn't otherwise be able to. On the low end sometimes we just need 7 extra bytes in a small function to fit in some extra bit of functionality. On the high end JL has drastically increased the size of XGStrategyAI.AIAddNewObjectives in an attempt to make the strategy AI a bit more interesting. He resized the function by 0x142D = 5165 bytes. The original size was 0x1BF=447 bytes and the new size is 0x15EC = 5612 bytes. Quite an increase :smile:

 

-------------------

 

One topic I wanted to bring up is that we seem to be experiencing a lot more glitchiness when trying to use local variables from other functions or classes. Sometimes it works fine, but sometimes it doesn't work at all. Sometimes it results in CTD, sometimes the value of the local variable appears to "get lost".

 

So, we've starting to make more use of a feature of UPKmodder (I'm not sure if wghost's tools support this, but they probably do ^_^) -- changing the type of a variable.

 

I've written up a small blurb on how to use UPKmodder to change a variable type:

 

-----

 

How to change the type of return values, function parameters, local variables and class variables.

 

All four of these objects work similarly with respect to changing their type.

 

The basic UPKmodder header format is :

OBJECT_ENTRY=bWuss@MarkWussySoldiers@XGFacility_Barracks
ACTION=typechange

OBJECT_ENTRY= is functionally identical to FUNCTION=. UPKmodder allows the alternate usage to assist with clarity.

 

The ACTION= line specifies that this is a typechange action. Typechanges are different from "standard" hex swaps because they alter the object table entry instead of the object itself.

 

The body format is quite simple:

[BEFORE_HEX]
OBJECT_TYPE=Core:BoolProperty@Core
SIZE=28 // hexadecimal
[/BEFORE_HEX]

[AFTER_HEX]
OBJECT_TYPE=Core:IntProperty@Core
SIZE=28 // hexadecimal
[/AFTER_HEX]

Only 2 parameters are needed, and both are required.

 

The OBJECT_TYPE must resolve to a reference in the import table. This is how the upk defines variable types. Using the name makes the change more patch-safe.

 

Valid types are:

• BoolProperty
• ByteProperty
• IntProperty
• FloatProperty
• ObjectProperty
• ClassProperty
• StrProperty
• StructProperty
• NameProperty
• InterfaceProperty

For the most part I'd expect bool, int, float, byte, and object to be the most commonly used.

 

Different objects can have different sizes, which can require adjusting the specified size. Increasing the size will not resize the upk -- it is primarily used to specify a smaller size when converting objects to simpler variables, which leaves some dead space. I don't know the size of all of the types yet, but :

• BoolProperty : x28 bytes
• ByteProperty : x28 bytes
• IntProperty : x28 bytes
• FloatProperty : x28 bytes
• ObjectProperty : x2C bytes

To swap between the first 4, simply put the size the same in BEFORE and AFTER. ObjectProperties are 4 bytes larger, as the last 4 bytes of the referenced object are a reference to the particular class the object references. Changing from a simpler type to an object is currently not possible, but an object variable can be converted to a simpler type with :

OBJECT_ENTRY=kCanvas@DebugAnims@XGUnit
ACTION=typechange

[BEFORE_HEX]
OBJECT_TYPE=Core:ObjectProperty@Core
SIZE=2C // hexadecimal
[/BEFORE_HEX]

[AFTER_HEX]
OBJECT_TYPE=Core:IntProperty@Core
SIZE=28 // hexadecimal
[/AFTER_HEX]

Note that the BEFORE is an ObjectProperty with listed size of 2C, and the AFTER is an IntProperty with size 28. This creates a small 4 byte "dead space" between the kCanvas object and the next object in the upk.

 

 

Below are a couple of working examples of typechanges I've created while working on Long War EW to use as a template:

 

 

Changing an object type parameter into an int type

 

 

MODFILEVERSION=4
UPKFILE=XComGame.upk 
GUID=5B 06 B8 18 67 22 12 44 85 9B A8 5B 9D 57 1D 4B // XComGame_EW_patch1.upk
OBJECT_ENTRY=kCanvas@DebugAnims@XGUnit
ACTION=typechange

//change object type parameter into int type

[BEFORE_HEX]
OBJECT_TYPE=Core:ObjectProperty@Core
SIZE=2C // hexadecimal
[/BEFORE_HEX]

[AFTER_HEX]
OBJECT_TYPE=Core:IntProperty@Core
SIZE=28 // hexadecimal
[/AFTER_HEX]
 

 

 

 

 

Changing a boolean local variable into an int variable:

 

 

MODFILEVERSION=4
UPKFILE=XComStrategyGame.upk
GUID=31 9C 3B 3F 9C 5D E4 40 AB AF 92 8E 25 65 74 F2 // XComStrategyGame_EW_patch1.upk
OBJECT_ENTRY=bWuss@MarkWussySoldiers@XGFacility_Barracks
ACTION=typechange

//change variable type from boolean to integer for loop over perks

[BEFORE_HEX]
OBJECT_TYPE=Core:BoolProperty@Core
SIZE=28 // hexadecimal
[/BEFORE_HEX]

[AFTER_HEX]
OBJECT_TYPE=Core:IntProperty@Core
SIZE=28 // hexadecimal
[/AFTER_HEX] 

 

 

[XMarksTheSpot]

Neat stuff, as always :smile:

 

I wonder, if modifying the object list is a possibility, what's stopping you from introducing new local variables? Or to put it differently, what modifications are needed to create new variables (or structs, functions, classes, etc. for that matter)?

By re-using an existing namelist entry (for the sake of simplicity) wouldn't you only need to append a new objectlist entry to the header? You'd likely need to adjust the file offsets of all objects to account for the extra entry size, but similar to the function resize operation that can be automated easily.

 

Or as another alternative, how about re-locating local variables by changing their owner? Assume you re-write a function to not use a certain local variable anymore (maybe some debug function) could it be moved to another function instead? Maybe some 'borrowed' locals don't work properly because of some internal memory management shenanigans by the engine which could be circumvented by re-locating the variable.

 

Just throwing around random ideas here :smile:

 

X.

Edited February 3, 2014 by XMarksTheSpot

[wghost81]

Is is possible to add new variables or even entire classes, but as I understand, Amineri don't want to use untested stuff. :wink:

 

But, Amineri, since you already expanding functions in-place and it's a big change to package structure, you may as well consider adding new objects. It won't change existing references, only offsets will be changed, but they do anyway with in-place expanding.

[Amineri]

I realize it's definitely within our capabilities now to add new objects.

 

As you both correctly point out, simply adding the new object to the end of the object list (resizing the upk and adjusting the objectlist references in a manner similar to resizing an object) would do the basic trick. In addition the "proto-header" would have to be adjusted to reflect the increased size of the objectlist table.

 

The main thing I haven't internalized (but is provided in wghost's link) is just how the linking of local, return value and parameters is done. I'm sure I can figure it out, it's just that I haven't done it yet. At some point I had to stop working on the tool and start actually using the tool to work on the Long War EW mod :smile:

 

I pretty much spent all of December struggling (with XMTS's tremendous help with the GUI elements) to get UPKmodder coded up, and now have spent the bulk of my free time in January porting/coding a bunch of stuff for Long War EW. My current upk_mod file count for Long War stands at 494 files. This includes function changes as well as variable modifications as above, as well as a few functions that have been deprecated for one reason or another, but there's still a lot of changes.

 

For example, I had to mod UISoldierPromotion.UpdateAbilityData for the basic Expanded Perk Tree, but then make further changes to support the psionics rework. I'm keeping the basic upk_mod file for the Expanded Perk Tree (so that I can apply just that mod without the psionics). So there's 3 versions of hex : vanilla, Expanded Perk Tree, and Expanded Perk Tree + Psionics, which results in 2 different upk_mod files.

 

Starting to close in on getting a releasable Long War EW mod, at which point I'll start fixing some backlogged bugs (e.g. a line ending with a null string doesn't parse) and adding new features, and being able to add new variables is definitely on the list.

 

-------------------

 

Regarding the adding of new classes, in a PM from amelocik he made the suggestion that for creating new classes we could use the existing Unreal Development Kit to develop stand-alone unreal package files, and then add import references to the new upk to the existing XComGame.upk and XComStrategyGame.upk. It might be simpler to add new classes that way then going through all of the steps needed to add enough objects to create a new class, default class, class variables, and class methods into an already cooked upk.

Edited February 3, 2014 by Amineri