R&D - New Modding Tools

[wghost81]

If user tries to install two mods, which are overwriting the same function A, he is in trouble anyway. :smile: You want to prevent user from installing incompatible mods - that's probably good.

 

Right now uninstaller in PatchUPK makes a backup copy of data chunks it overwrites. Suppose, user installs mod A and modify function A and then mod B, which modifies the same function. Then he discovers that game is not working. He uses uninstall script for mod B and revert to perfectly working mod A, as uninstaller restores upk files to their previous state. Mods are uninstalled completely, I can't see the problem here. Of course, with overlapping changes if user will try to uninstall mod A with mod B changes still present, he may break the game. But that's what the backups are for. :smile: He may always revert to clean game by using very first backup files and start again. :smile:

 

In case of compatible mods user can install/uninstall them in any order. In case there are some dependences, mod readme should contain installation details and compatibility info.

Edited December 29, 2013 by wghost81

[Amineri]

For those testing it out, a small patch (bugs helped found courtesy of johnnylump) on UPKmodder is up, v0.71:

 

v 0.71 (r118)

Fixed bug causing incorrect file and memory sizes reported in tree view

Fixed bug causing "test file status" to fail when AFTER hex was installed

Fixed graphical bug that caused last hex token to not be fully highlighted

[wghost81]

UPD:

Understood your problem! Again, it's created by before-after style patching. :wink: If installer makes backup of current data and then uses it to revert to previous saved version, rather to some pre-defined (by before hex) vanilla code - everything will be fine.

[Amineri]

For end-use patching it is definitely possible to read and store a copy of the previous state. Basically a simple sort of "undo" operation. I didn't realize that PatchUPK had that functionality, and that is really quite excellent!

 

I agree 100% that managing your backups this way to allow for un-install capabilities is the way to go.

 

For development (and mid/large-scale development in particular) I haven't figured out a good way to handle this. Hence JL and I still primarily rely on before-after style to manage the back and forth of constantly churning changes.

 

 

 

If user tries to install two mods, which are overwriting the same function A, he is in trouble anyway. :smile: You want to prevent user from installing incompatible mods - that's probably good.

 

In some cases this type of thing is almost unavoidable, as there are some popular and very large functions that can potentially have multiple mods applied to them and still be compatible.

 

As long as each of the changes is made to just a few lines (not overwriting the entire function for the sake of changing a couple of lines), and if the lines changes are in different parts of the function, these changes can be made to the same function and be compatible. It's precisely for this reason that I try to minimize the number of lines of code that I am changing -- in order to maximize the likelihood of compatibility.

 

It's really akin to making minimal changes in 'regular' code in order to make the version-controlled merging process as painless as possible. Sometimes it doesn't work out that way, but sometimes it does.

 

It's often the case that JL and I end up modding different parts of the same function. XGUnit.OnTakeDamage is a prime example.

The XGUnit.OnTakeDamage function is on the order of ~425 lines, so there is a decent amount of room for several mods to make changes in distinct parts.

 

For example one mod might be altering the HEAT Ammo damage mechanic at the front of the function (lines 23-30), while another could be altering the shield HP mechanic (lines 61-66), while yet a third could be altering the stunning mechanics (lines 120-174).

[wghost81]

Good example. But anyway, I still think that multiple changes to one function by different people is a dangerous thing. And mod author should provide compatibility list in this case.

 

Version control in case of XCOM modding is a real pain. I agree this is a difficult situation. But it is still possible to work on separate exported binaries (you can use UE Explorer (or FindObjectEntry), it gives owner.owner...name.type names to exported binaries which is very helpful) and merge your changes to separate binary which can be then checked and imported back to upk.

 

BTW, to check my code and recalculate offsets/sizes I use "junk" (temporary) upk files. :smile: This may not be possible with large mods with lots of renamed variables, but your tool is almost as good as decompiler and you can use it instead. Ability to decompile separate scripts "in context" of given upk is something I miss in UE Explorer...

[Amineri]

 

 

Good example. But anyway, I still think that multiple changes to one function by different people is a dangerous thing. And mod author should provide compatibility list in this case.

 

In an ideal world each mod author would test her mod for compatibility against every other released mod, but in this world I don't think that's every happened. We get questions for Long War all the time about "why does ToolBoks feature X or Y not work with Long War?" In some cases we can't answer, because we're not even sure what feature Y is doing.

 

 

 

BTW, to check my code and recalculate offsets/sizes I use "junk" (temporary) upk files. :smile: This may not be possible with large mods with lots of renamed variables, but your tool is almost as good as decompiler and you can use it instead. Ability to decompile separate scripts "in context" of given upk is something I miss in UE Explorer...

 

What I used to do is apply the "bad" hex change to my working upk, open it in UE Explorer, get the data and fix the offsets (either manually before Bertilsson's tool, or using his tool after he got it working). I'd then create revert the upk back to the vanilla state and replace my "after" hex with the updated block provided by the tool.

 

I've now added the ability to compute memory positions/sizes directly into UPKmodder so it displays the same information as available in UE Explorer's Tokens view. Also every jump offset position is colored to make them really easy to spot. With that I'm able to correct the offsets quite quickly by hand before ever having to apply anything to the upk. Ultimately I'd like to incorporate the same algorithms as in Bertilsson's tool directly into UPKmodder.

 

Because we're now using the variable names to allow updating from one patch version to another, we've discontinued renaming of variables. I'm trying to think if there's a way to do it and be able to easily update.

 

All of the info is there in UPKmodder to decompile (I've decompiled other's code by hand purely within it). Completely automating it is a bit more work, but would allow some nice options such as "test decompile hex" to test out some code you've just written, as well as "direct import function from upk". I'm not planning on adding any sort of general browsing capability, as there's a lot of extra data structures to manage and UE Explorer already does that job perfectly.

[Bertilsson]

Because we're now using the variable names to allow updating from one patch version to another, we've discontinued renaming of variables. I'm trying to think if there's a way to do it and be able to easily update.

I think the key to this may be found while figuring out a good way to handle WGhost's latest discoveries.

Instead of renaming an object, create a new object with proper name and use that object instead.

 

Edit:

Something like this might work:

NEWOBJECT=XGFacility_Barracks.PickAClass.NewVarName;Type

 

After that this new object can be referenced in the same was as any other object.

 

Edit 2: Or even better:

MAKEOBJECT=XGFacility_Barracks.PickAClass.NewVarName;Type

 

If the object does not already exist: Create the object.

If the object is of different type: Change type

Edited December 29, 2013 by Bertilsson

[Drakous79]

File backup and backup of overwritten data chunks is cool. Have to agree, two mods modding the same values (hex, ini, lanfuage strings) will never work together.

 

Maybe we could create some compatibility chart on Nexus Wiki with info about altered objects. People can check, if an object, they plan to work with, has been already altered by others. Then they could check the mod for compatibility or PM authors with questions.

// Name                How it was altered    Mod's name    Authors
XGUnit.OnTakeDamage    Rewritten             LW            Amineri / johnnylump
Some.Function          Expanded              XYZ           DarthVader

There can be links to description of changes.

 

But checking user's installation may still be useful, not just for prompts like: "There's problem. Install anyway?", but to avoid situations like patching expanded function with relative offset and data belonging to unexpanded function.

 

Thought abut MD5 - is shorter than some hex chunks.

 

MD5 of the function is as expected - no problem

MD5 is different, so what's wrong?

- function's size is bigger/smaller - problem (expanded/shrinked)

- function's virtual size is bigger/smaller - problem (content changed)

- function's virtual size is equal - problem (content changed, but may be compatible)

[wghost81]

If the object does not already exist: Create the object.

If the object is of different type: Change type

Somehow I don't like this kind of automation. May be it's my suspicious nature. :smile:

 

Auto-creating new objects may cause a problem if there is, say, typo in object/type name. That's why I prefer explicit commands instead of implicit hidden operations. :smile:

[Bertilsson]

How about breaking them down into two separate commands then?

 

CreateObject

If the object does not exist: Create the object.

If the object exist and is of correct type: Great! No action needed.

If the object exist but is of incorrect type: Something is wrong, throw error to alert user that he/she may accidentally be up to no good.

 

SetType

If object is not of requested type: Change type

If object is already of requested type: Great! No action needed.

 

The upside to not see target already achieved as a problem is repeat-ability.

I can change an unrelated detail and run the same script again and again.